go / s3

updated July 17, 2026

The AWS SDK for Go is the strongest case for a hand-rolled client. It's a large dependency tree to call three S3 operations: list objects, get an object, and presign a PUT. The only real work the SDK saves is request signing, and AWS Signature Version 4 is a few hundred lines.

The client

s3.Client implements only the three operations I use. Two HTTP clients: a 10s-timeout one for small request/response calls, and a no-timeout one for streaming downloads that callers may read for much longer:

type Client struct {
	region     string
	signer     *SigV4
	httpClient *http.Client
	// streamClient is separate from httpClient because GetObject returns
	// a streaming body that callers may read for much longer than the
	// 10s timeout we use for small request/response calls.
	streamClient *http.Client
	// endpointFn is a hook so tests can route requests to httptest.
	endpointFn func(bucket string) string
}

func NewClient(region, accessKeyID, secretAccessKey string) (*Client, error) {
	signer, err := NewSigV4("s3", region, accessKeyID, secretAccessKey)
	if err != nil {
		return nil, err
	}
	return &Client{
		region:       region,
		signer:       signer,
		httpClient:   &http.Client{Timeout: 10 * time.Second},
		streamClient: &http.Client{},
		endpointFn: func(bucket string) string {
			return fmt.Sprintf("https://%s.s3.%s.amazonaws.com", bucket, region)
		},
	}, nil
}

SigV4

The signer supports two modes: header-signing for GET requests and query-string presigning for PUT URLs. Session tokens, asymmetric signatures, event streams, and path normalization are all in the spec and none are implemented, because I don't use them.

Construction validates every field, so a misconfigured worker fails at startup instead of producing signatures S3 silently rejects:

func NewSigV4(service, region, accessKeyID, secretAccessKey string) (*SigV4, error) {
	// ... trim each field
	if service == "" || region == "" || accessKeyID == "" || secretAccessKey == "" {
		return nil, errors.New("s3: missing service, region, access_key_id, or secret_access_key")
	}
	// ...
}

Signing builds the canonical request, derives the string to sign, and computes the signature. The canonical request is where SigV4 is exacting: method, path, sorted query, sorted headers, signed header list, and payload hash, each newline-joined:

func (s *SigV4) canonicalRequest(method string, u *url.URL, headers map[string]string, contentSHA string) string {
	return strings.Join([]string{
		strings.ToUpper(method),
		canonicalPath(u),
		canonicalQuery(u.RawQuery),
		canonicalHeaders(headers) + "\n",
		signedHeaders(headers),
		contentSHA,
	}, "\n")
}

The signing key is a chain of HMACs over date, region, service, and a fixed terminator:

func (s *SigV4) signature(date, stringToSign string) string {
	kDate := hmacSHA256([]byte("AWS4"+s.secretAccessKey), date)
	kRegion := hmacSHA256(kDate, s.region)
	kService := hmacSHA256(kRegion, s.service)
	kSigning := hmacSHA256(kService, "aws4_request")
	return hex.EncodeToString(hmacSHA256(kSigning, stringToSign))
}

Escaping is the trap

SigV4 wants RFC 3986 escaping that differs from Go's stdlib in two spots: space is %20, not +, and ~ stays unescaped. Get this wrong and the signature mismatches with an opaque 403:

func escape(s string) string {
	escaped := url.QueryEscape(s)
	escaped = strings.ReplaceAll(escaped, "+", "%20")
	escaped = strings.ReplaceAll(escaped, "%7E", "~")
	return escaped
}

Presigned PUT

PresignedPutURL makes no HTTP call. It signs a URL the uploader can PUT to directly. Any headers signed here (like Content-Type) must be sent verbatim by the uploader or the signature won't validate:

func (c *Client) PresignedPutURL(bucket, key, contentType string, expiresIn time.Duration) (string, error) {
	return c.signer.PresignURL(http.MethodPut, c.objectURL(bucket, key), expiresIn, map[string]string{
		"Content-Type": contentType,
	}, time.Now())
}

Presigning puts the auth in query params (X-Amz-Signature and friends) with UNSIGNED-PAYLOAD as the content hash, since the signer never sees the bytes the client will upload.

Two retry paths

ListObjectsV2 buffers a small XML body, so it goes through the shared backoff helper. The build closure re-signs per attempt, because the signature binds to x-amz-date and a retry crosses a second boundary:

func (c *Client) getBytes(ctx context.Context, url string) ([]byte, error) {
	build := func() (*http.Request, error) {
		signed, err := c.signer.SignRequest(http.MethodGet, url, nil, nil, time.Now())
		if err != nil {
			return nil, fmt.Errorf("sign: %w", err)
		}
		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
		if err != nil {
			return nil, err
		}
		for k, v := range signed {
			req.Header.Set(k, v)
		}
		return req, nil
	}

	res, err := httputil.Do(ctx, build, httputil.Config{
		Client:         c.httpClient,
		TransientCodes: transientCodes,
	})
	// ... check status, return res.Body
}

GetObject can't use the helper: httputil.Do consumes the response body, but a streaming download must hand the open body back to the caller. So it keeps its own retry loop, still reusing httputil.DefaultRetryDelays and httputil.Sleep so tests share the same zero-delay knob:

func (c *Client) GetObject(ctx context.Context, bucket, key string) (io.ReadCloser, error) {
	u := c.objectURL(bucket, key)
	var lastErr error
	for i, delay := range httputil.DefaultRetryDelays {
		// ... re-sign, build request
		resp, err := c.streamClient.Do(req)
		// ... on transient error or transient code, sleep and continue
		if resp.StatusCode/100 != 2 {
			resp.Body.Close()
			return nil, fmt.Errorf("HTTP %d", resp.StatusCode)
		}
		return resp.Body, nil // caller closes
	}
	// ... return lastErr
}

The caller closes the returned ReadCloser. That single ownership difference is why streaming needs its own loop.

Tests

endpointFn routes requests to an httptest.NewServer, so the whole client (signing included) runs without AWS. The signer has its own tests that pin the signature against known inputs, so a stray escaping change fails loudly instead of in production with a 403.

← All articles