AWS S3 Tips

aws s3 java

AWS S3 Commands

Command	Description
aws s3 mb	Create a new S3 bucket
aws s3 ls	List the contents of a bucket or list all buckets
aws s3 cp	Copy objects between buckets or from a local directory to a bucket
aws s3 mv	Move objects between buckets or rename objects within a bucket
aws s3 rm	Delete objects from a bucket
aws s3 sync	Sync a local directory with a bucket or vice versa
aws s3 presign	Generate a pre-signed URL for accessing an object in a bucket
aws s3 s3api	Low-level API commands - Perform more advanced operations like multipart uploads and bucket lifecycle management

Create a Bucket

aws s3 mb s3://my-new-bucket

List Buckets and Objects

aws s3 ls
aws s3 ls s3://my-bucket

Copy Objects

aws s3 cp myfile.txt s3://my-bucket/myfile.txt
aws s3 cp s3://my-bucket/myfile.txt myfile.txt

Move Objects

aws s3 mv s3://my-bucket/oldname.txt s3://my-bucket/newname.txt
aws s3 mv myfile.txt s3://my-bucket/myfile.txt

Delete Objects

aws s3 rm s3://my-bucket/myfile.txt
aws s3 rm s3://my-bucket --recursive --exclude "*" --include "*.txt"

Synchronize Directories

aws s3 sync my-local-directory s3://my-bucket
aws s3 sync s3://my-bucket my-local-directory

Generate Pre-signed URLs

# To generate a pre-signed URL for an object, valid for 1 hour (3600 seconds):
aws s3 presign s3://my-bucket/myfile.txt --expires-in 3600

Using aws s3api for Advanced Operations

aws s3api create-multipart-upload --bucket my-bucket --key my-large-file
aws s3api list-parts --bucket my-bucket --key my-large-file --upload-id <UploadId>
aws s3api complete-multipart-upload --bucket my-bucket --key my-large-file --upload-id <UploadId> --multipart-upload file://parts.json

Common S3 APIs

API Operation	Description
CreateBucket	Creates a new S3 Bucket
DeleteBucket	Deletes a S3 bucket
PutObject	Uploads an object to a bucket
GetObject	Retrieves an objects from a bucket
DeleteObject	Deletes an object from a bucket
ListObjects	Lists objects in a bucket
CopyObject	Copies an object from one bucket to another
MoveObject	Moves an object within a bucket or renames it
MultipartUpload	Initiates, uploads parts, and completes a multipart upload
ListParts	List parts of a multipart upload
CompleteMultipartUpload	Completes a multipart upload
PutObjectAcl	Sets the access control list (ACL) for an object
GetBucketAcl	Retrieves the ACL for a bucket
PutBucketAcl	Sets the ACL for a bucket
GetBucketPolicy	Retrieves the bucket policy
PutBucketPolicy	Sets the bucket policy

Processing a Json file in Java

Dependencies

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.29.0</version>
</dependency>
<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.12.3</version>
</dependency>

Java code

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;

public class S3JsonModifier {
    public static void main(String[] args) throws IOException {
        Region region = Region.US_EAST_1;
        S3Client s3 = S3Client.builder()
            .region(region)
            .build();

        String bucketName = "your-bucket-name";
        String key = "your-file-key.json";
        Path tempFilePath = Paths.get("temp.json");

        // Download JSON file from S3
        GetObjectRequest getObjectRequest = GetObjectRequest.builder()
            .bucket(bucketName)
            .key(key)
            .build();
        s3.getObject(getObjectRequest, tempFilePath);

        // Read and modify JSON
        ObjectMapper objectMapper = new ObjectMapper();
        JsonNode jsonNode = objectMapper.readTree(Files.newBufferedReader(tempFilePath));
        if (jsonNode.isObject()) {
            ((ObjectNode) jsonNode).put("newField", "newValue");
        }

        // Write modified JSON to file
        objectMapper.writeValue(tempFilePath.toFile(), jsonNode);

        // Upload modified JSON back to S3
        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
            .bucket(bucketName)
            .key(key)
            .build();
        s3.putObject(putObjectRequest, RequestBody.fromFile(tempFilePath));

        // Clean up temporary file
        Files.deleteIfExists(tempFilePath);

        System.out.println("JSON file modified and uploaded successfully!");
    }
}

Written on November 11, 2024