aws sdk cpp中S3相关的应用和示例

Posted on | In | |

概述

aws提供多种sdk去访问S3，包括java、go、php、js、ruby、net、c++等，本篇文章结合作者最近应用的实践，介绍aws sdk cpp中访问S3的使用，示例中包括对S3的基本put，get等操作。

在使用aws sdk cpp中，发现它还不是那么完善，很多对S3的操作都找不到示例，并且github上这部分的源码更新很快，估计也是在持续开发和完善中，建议读者在自己应用时，下载最新的sdk版本后测试。

安装

测试系统：CentOS Linux release 7.2.1511

  1. 下载github上的sdk源码

    1
    # git clone https://github.com/aws/aws-sdk-cpp

    或者直接下载zip压缩包后解压缩

  2. 安装依赖包

    1
    2
    3
    4
    5
    yum install -y gcc-c++
    yum install -y cmake
    yum install -y zlib-devel
    yum install -y openssl-devel
    yum install -y curl-devel

    Ubuntu系统上：

    1
    2
    3
    4
    5
    6
    apt-get install -y g++
    apt-get install -y cmake
    apt-get install -y libssl-dev
    apt-get install -y libcurl4-openssl-dev
    apt-get install -y uuid-dev
    apt-get install -y libboost-all-dev

  3. 安装aws cpp sdk

    1
    2
    3
    4
    5
    6
    7
    8
    9
    mkdir build_dir
    cd build_dir
    cmake -DCMAKE_BUILD_TYPE=Release ../aws-sdk-cpp

    # just make and install core and s3
    make -j `nproc` -C aws-cpp-sdk-core
    make -j `nproc` -C aws-cpp-sdk-s3
    make install -C aws-cpp-sdk-core
    make install -C aws-cpp-sdk-s3

示例

基本代码

要使用aws c++ sdk，必须包含如下基本代码

1
2
3
4
5
6
7
8
9
10
11
#include <aws/core/Aws.h>
int main(int argc, char** argv)
{
   Aws::SDKOptions options;
   Aws::InitAPI(options);
   {
      // make your SDK calls here.
   }
   Aws::ShutdownAPI(options);
   return 0;
}

使用aws s3 client来访问对象存储

1
2
3
4
5
6
7
8
9
10
Aws::Client::ClientConfiguration cfg;
cfg.endpointOverride = "your s3 endpoint";
cfg.scheme = Aws::Http::Scheme::HTTP;
cfg.connectTimeoutMs = 100000;
cfg.requestTimeoutMs = 100000;

Aws::Auth::AWSCredentials cred("your access key", "your secret key");
S3Client client(cred, cfg, false, false);

// Then you can use this client object to access s3 object

示例1 - list bucket

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
cat aws-s3-bucket.cpp
#include <iostream>
#include <aws/s3/S3Client.h>
#include <aws/core/Aws.h>
#include <aws/core/auth/AWSCredentialsProvider.h>

using namespace Aws::S3;
using namespace Aws::S3::Model;
using namespace std;

int main(int argc, char* argv[])
{
    Aws::SDKOptions options;
    Aws::InitAPI(options);

    Aws::Client::ClientConfiguration cfg;
    cfg.endpointOverride = "...";
    cfg.scheme = Aws::Http::Scheme::HTTP;

    Aws::Auth::AWSCredentials cred("...", "...");
    S3Client client(cred, cfg, false, false);

    auto response = client.ListBuckets();
    if (response.IsSuccess()) {
        auto buckets = response.GetResult().GetBuckets();
        for (auto iter = buckets.begin(); iter != buckets.end(); ++iter) {
            cout << iter->GetName() << "\t"
                 << iter->GetCreationDate().ToLocalTimeString(Aws::Utils::DateFormat::ISO_8601) << endl;
        }
    } else {
        cout << "Error while ListBuckets " << response.GetError().GetExceptionName()
            << " " << response.GetError().GetMessage() << endl;
    }

    Aws::ShutdownAPI(options);

    return 0;
}

示例2 - get object

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
/*
 * Get object range content to local file
 */
int S3Storage::get_object_to_file(const S3Client &client,
                    				 const Aws::String bucket,
                               	 const Aws::String object,
                                 	 const string &file_name,
                               	 const uint64_t offset,
                       			  	 uint64_t length)
{
...
    GetObjectRequest request;
    request.WithBucket(bucket).WithKey(object);

    string range = S3_RANGE_STRING(offset, length);
    request.SetRange(range.c_str());

    auto outcome = client.GetObject(request);
    if(!outcome.IsSuccess()) {
        cout << "GetObject error: " <<
            outcome.GetError().GetExceptionName() << " - " <<
            outcome.GetError().GetMessage() << endl;
        return -1;
    }

    Aws::OFStream local_file;
    local_file.open(file_name.c_str(), std::ios::in | std::ios::out | std::ios::binary);
    assert(local_file.good());
    local_file.seekp(offset, std::ios::beg);
    local_file << outcome.GetResult().GetBody().rdbuf();
    local_file.close();
...
}

示例2 - put object

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
/*
 * Put whole file to s3 object
 */
int S3Storage::put_object_from_file(const S3Client &client,
										   const Aws::String bucket,
                                    const Aws::String object,
                                    const string &file_name)
{
...
    PutObjectRequest request;
    request.WithKey(object).WithBucket(bucket);

    auto input_data = Aws::MakeShared<Aws::FStream>("PutObjectInputStream",
                file_name.c_str(), std::ios_base::in | std::ios_base::binary);

    //set the stream that will be put to s3
    request.SetBody(input_data);

    auto outcome = client.PutObject(request);
    if(outcome.IsSuccess()) {
        cout << "Put object succeeded!" << endl;
    } else {
        cout << "PutObject error: " <<
             outcome.GetError().GetExceptionName() << " - " <<
             outcome.GetError().GetMessage() << endl;
    }
...
}

因有需要读取文件的一部分内容，然后上传到S3上的一个object，查询了好多资料没找到示例，最后翻阅aws cpp sdk的源码，参考相关的实现，花费了很多时间才试验出来，贴给有相同需要的朋友。。。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
/*
 * Put one file slice to s3 object
 */
 int S3Storage::put_object_from_file_slice(const S3Client &client,
										          const Aws::String bucket,
                                           const Aws::String object,
                                           const string &file_name,
                                           const uint64_t offset,
                                           const uint64_t length)
{
...
    // Upload file with slices to s3 paralleled
    PutObjectRequest request;
    request.WithKey(object).WithBucket(bucket);

    Aws::FStream local_file;
    local_file.open(file_name.c_str(), std::ios::in | std::ios::binary);
    assert(local_file.good());

    Array<uint8_t> file_contents(length);
    local_file.seekg(offset, std::ios::beg);
    cout << "file read offset " << local_file.tellg() << endl;
    local_file.read((char*)file_contents.GetUnderlyingData(),
                    file_contents.GetLength());
    local_file.close();

    //set the stream that will be put to s3
    auto sbuff = Aws::New<Aws::Utils::Stream::PreallocatedStreamBuf>(CLASS_TAG,
                                    &file_contents, file_contents.GetLength());
    auto sbody = Aws::MakeShared<Aws::IOStream>("whatever string", sbuff);
    request.SetBody(sbody);

    auto outcome = client.PutObject(request);
    if(outcome.IsSuccess()) {
        cout << "Put object succeeded!" << endl;
    } else {
        cout << "PutObject error: " <<
                outcome.GetError().GetExceptionName() << " - " <<
                outcome.GetError().GetMessage() << endl;
    }
...
}

参考

http://docs.aws.amazon.com/zh_cn/sdk-for-cpp/v1/developer-guide/welcome.html
https://github.com/aws/aws-sdk-cpp
http://alientechlab.com/aws-sdk-cpp-part-1/
http://www.cnblogs.com/qiuyi21/p/7239129.html

支持原创