AWS S3 or Amazon S3 is a storage system from Amazon to store and retrieve files from anywhere on the web. It is a highly scalable, reliable, fast, inexpensive data storage system from Amazon. It provides easy to use developer kit to store and retrieve files.
In this tutorial you will learn how to upload bulk files to S3 bucket aws golang. We will use the Go AWS SDK to upload files to AWS S3.
The AWS SDK for Go requires Go 1.15 or later versions. You can view your current version of Go by running the following command.
go version
For information about installing or upgrading your version of Go, see https://golang.org/doc/install
Before you can use the AWS SDK for Go V2, you must have an Amazon account. See How do I create and activate a new AWS account? for details
Download the AWS Golang SDK for using in your codebase using below command
go get github.com/aws/aws-sdk-go/aws
We will cover this tutorial step by step to upload bulk files to AWS S3 Bucket server using Golang with example code.
Load require packages
We will create main.go file and import necessary packages we need to upload the files into AWS S3 Bucket.
package main
import (
"bytes"
"fmt"
"log"
"net/http"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)Custom struct for receiver
We will be using a Session struct so we can use same it as receiver, and we will pass this receiver to our main upload function,
Via using a custom struct we’re eliminating the duplication from our code. We’ll be using only single session for our all bulk files to upload into s3 bucket aws golang.
type Session struct {
S3Session *session.Session
}Set up main configuration and AWS session
We will setup configuration using AWS S3 REGION and SECRET ID and SECRET KEY and create single AWS session to upload multiple files to AWS S3. Then we will call method upload() and pass AWS session instance and file details to upload file to AWS S3 server.
func main() {
paths := []string{"", ""}
credential := credentials.NewStaticCredentials(
os.Getenv("SECRET_ID"),
os.Getenv("SECRET_KEY"),
"",
)
awsConfig := aws.Config{
Region: aws.String(os.Getenv("REGION")),
Credentials: credential,
}
s, err := session.NewSession(&awsConfig)
if err != nil {
log.Println("failed to create S3 session:", err.Error())
return
}
se := Session{s}
err = se.upload(paths)
if err != nil {
log.Println(err.Error())
return
}
}
As you can see in the above code we are first loading our all config credentials using os.Getenv() and passing them to aws sdk function credentials.NewStaticCredentials() and also we’re calling aws.Config struct of aws and passing credential variable in that struct and also loading REGION value from environment variable.
Then we’re passing awsConfig of type aws.Config into session.NewSession(), which will return us the aws s3 session so we can use single session to upload multiple files, if anything wrong happens on the time of session creation it’ll print the custom message alongside of err.Error() returned by session.NewSession().
In the last we’re passing session into our Session struct we created earlier in this tutorial, after that we’re calling the upload method and passing all paths of files needs to upload and session information as function receiver.
Create Upload Method to Upload Bulk Files to AWS S3
We will create method upload() to upload files to AWS S3 server. We will open the file one by one and store into buffer and then put the file to AWS S3 using PutObject() method from S3. We will also specify options in the PutObjectInput when uploading the file. We will also enable AES256 encryption on files using ServerSideEncryption options of s3.PutObjectInput struct.
func (s Session) upload(paths []string) error {
for _, path := range paths {
upFile, err := os.Open(path)
if err != nil {
log.Printf("failed %s, error: %v", path, err.Error())
continue
}
defer upFile.Close()
upFileInfo, err := upFile.Stat()
if err != nil {
log.Printf("failed to get stat %s, error: %v", path, err.Error())
continue
}
var fileSize int64 = upFileInfo.Size()
fileBuffer := make([]byte, fileSize)
upFile.Read(fileBuffer)
// uploading
_, err = s3.New(s.S3Session).PutObject(&s3.PutObjectInput{
Bucket: aws.String(os.Getenv("BUCKET_NAME")),
Key: aws.String(path),
ACL: aws.String("public-read"), // could be private if you want it to be access by only authorized users
Body: bytes.NewReader(fileBuffer),
ContentLength: aws.Int64(int64(fileSize)),
ContentType: aws.String(http.DetectContentType(fileBuffer)),
ContentDisposition: aws.String("attachment"),
ServerSideEncryption: aws.String("AES256"),
StorageClass: aws.String("INTELLIGENT_TIERING"),
})
if err != nil {
log.Printf("failed to upload %s, error: %v", path, err.Error())
continue
}
url := "https://%s.s3-%s.amazonaws.com/%s"
url = fmt.Sprintf(url, os.Getenv("BUCKET_NAME"), os.Getenv("REGION"), path)
fmt.Printf("Uploaded File Url %s\n", url)
}
return nil
}
We have created a method called upload() which accepts the Session as receiver and Session struct have aws session which we created in main function, we are also passing all files which needs to be upload into upload() method.
In upload() method you can see in the beginning we’re a loop over paths variable and in the loop we’re first getting calling the os.Open then after opening file we’re calling the Stat method to get the all information about that opened file which we need to pass to aws s3 sdk methods.
As you can see we only required these mentioned path, fileSize and fileBuffer information’s to pass to s3.PutObjectInput so it can use to create a valid http request for uploading the files to aws s3 bucket.
Full Code to Upload Bulk Files to AWS S3 with Golang
Here is the complete code to upload the bulk files to AWS S3 server using Golang.
package main
import (
"bytes"
"fmt"
"log"
"net/http"
"os"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
type Session struct {
S3Session *session.Session
}
func main() {
paths := []string{"", ""}
credential := credentials.NewStaticCredentials(
os.Getenv("SECRET_ID"),
os.Getenv("SECRET_KEY"),
"",
)
awsConfig := aws.Config{
Region: aws.String(os.Getenv("REGION")),
Credentials: credential,
}
s, err := session.NewSession(&awsConfig)
if err != nil {
log.Println("failed to create S3 session:", err.Error())
}
se := Session{s}
err = se.upload(paths)
if err != nil {
log.Println(err.Error())
return
}
}
func (s Session) upload(paths []string) error {
for _, path := range paths {
upFile, err := os.Open(path)
if err != nil {
log.Printf("failed %s, error: %v", path, err.Error())
continue
}
defer upFile.Close()
upFileInfo, err := upFile.Stat()
if err != nil {
log.Printf("failed to get stat %s, error: %v", path, err.Error())
continue
}
var fileSize int64 = upFileInfo.Size()
fileBuffer := make([]byte, fileSize)
upFile.Read(fileBuffer)
// uploading
_, err = s3.New(s.S3Session).PutObject(&s3.PutObjectInput{
Bucket: aws.String(os.Getenv("BUCKET_NAME")),
Key: aws.String(path),
ACL: aws.String("public-read"), // could be private if you want it to be access by only authorized users
Body: bytes.NewReader(fileBuffer),
ContentLength: aws.Int64(int64(fileSize)),
ContentType: aws.String(http.DetectContentType(fileBuffer)),
ContentDisposition: aws.String("attachment"),
ServerSideEncryption: aws.String("AES256"),
StorageClass: aws.String("INTELLIGENT_TIERING"),
})
if err != nil {
log.Printf("failed to upload %s, error: %v", path, err.Error())
continue
}
url := "https://%s.s3-%s.amazonaws.com/%s"
url = fmt.Sprintf(url, os.Getenv("BUCKET_NAME"), os.Getenv("REGION"), path)
fmt.Printf("Uploaded File Url %s\n", url)
}
return nil
}
your comments are appreciated and if you wants to see your articles on this platform then please shoot a mail at this address kusingh@programmingeeksclub.com
Thanks for reading 🙂
Pingback: Golang Eğitimi: Golang Kullanarak Toplu Dosyaları aws S3’e Yükleme – Dandp
Pingback: Golang Tutorial: Add Bulk Information to aws S3 Utilizing Golang