DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Enterprise AI Trend Report: Gain insights on ethical AI, MLOps, generative AI, large language models, and much more.

2024 Cloud survey: Share your insights on microservices, containers, K8s, CI/CD, and DevOps (+ enter a $750 raffle!) for our Trend Reports.

PostgreSQL: Learn about the open-source RDBMS' advanced capabilities, core components, common commands and functions, and general DBA tasks.

AI Automation Essentials. Check out the latest Refcard on all things AI automation, including model training, data security, and more.

Related

  • Securing Cloud Storage Access: Approach to Limiting Document Access Attempts
  • API Governance: Ensuring Control and Compliance in the Era of Digital Transformation
  • 10 Commandments of API-First Development
  • Secure Your API With JWT: Kong OpenID Connect

Trending

  • Those Were The Days?! A Humorous Reflection on the Evolution of Software Engineering
  • Vector Tutorial: Conducting Similarity Search in Enterprise Data
  • How To Get Started With New Pattern Matching in Java 21
  • Service Mesh Unleashed: A Riveting Dive Into the Istio Framework
  1. DZone
  2. Software Design and Architecture
  3. Security
  4. How To Scan GCP Storage Files for Threats Using Go

How To Scan GCP Storage Files for Threats Using Go

This article discusses the importance of integrating external security solutions with cloud storage instances and provides a solution to integrate with GCP.

By 
Brian O'Neill user avatar
Brian O'Neill
DZone Core CORE ·
Jul. 10, 23 · Tutorial
Like (2)
Save
Tweet
Share
2.8K Views

Join the DZone community and get the full member experience.

Join For Free

As enterprise cloud storage solutions steadily gain momentum across global markets, the anti-virus and malware security policies deployed to protect these pay-per-scale services become more and more robust. Naturally, the taller the castle wall becomes, the higher the siege ladder is built to climb it, but the perpetual battle between threat actors and cloud security practitioners appears to march us ever closer toward a safer digital future.

That said, the catastrophic damage from successful cloud storage attacks cannot be understated, and the evolution of this attack vector should never be underestimated. Public, private, and hosted cloud storage services are all extremely worthwhile targets for threat actors, playing host to valuable data that moves frequently between user devices, and a variety of potent attack vectors are regularly explored to penetrate networks through these locations. Threat actors can leverage well-known techniques like phishing to goad users into saving and storing malicious content in a public location, and they can disguise malware very effectively in direct client upload scenarios through built-in file features like Office macros or PDF password-protection measures.  

If in-storage threat detection policies don’t perform scans quickly enough, or if they aren’t configured to flag disguised file threats, malicious files can be rapidly distributed to unsuspecting users who, upon opening them, unwittingly initiate sudden and unpredictable attacks. It’s critical to always view cloud storage security as a multi-policy effort, with more than one integrated security solution in place to increase the likelihood that file upload threats are detected rapidly after reaching a storage instance.

Google Cloud Platform (GCP) Storage, which occupies roughly 10-13% of the global cloud storage market share compared to its competitors, is one of many popular and fast-growing solutions which benefit from the integration of external threat detection policies. Much like its competitors, GCP makes it easy for developers to integrate additional security measures alongside its built-in security architecture. In the remainder of this article, I’ll demonstrate one free-to-use API solution which can be quickly integrated to incorporate a wide range of threat detection policies against file uploads in any given GCP bucket.

Demonstration

The cloud storage scanning API provided below is designed to integrate with a Google Cloud Storage Bucket and scan files using the basic information available in any GCP admin account. This solution is equipped to reference files against a frequently updated list of virus and malware signatures (currently totaling more than 17 million signatures, including trojans, ransomware, and spyware), and it’s also capable of detecting a variety of increasingly common hidden threat types through in-depth content verification. These hidden content threat types include executables, invalid files, scripts, password-protected files, macros, XML files (including XXE threats), Insecure Deserialization objects, and HTML input, and each one of these threats can be blocked or allowed by setting custom policies (using Boolean values in the API request body).  

For storage use cases involving direct user file uploads to GCP buckets, you can additionally set custom restrictions against unwanted file types by supplying a comma-separated list of accepted file extensions in the API request. When this policy is set, each file upload will be verified against your list of accepted file extensions.

To integrate this API with your GCP bucket, you’ll need to set the following information in your requests:

  1. Bucket name: The name of your bucket in GCP storage
  2. Object name: The name of the objects or files in GCP storage
  3. JSON credential file: The Service Account credential for Google Cloud, stored in a JSON file

Files that contain virus or malware signatures and/or violate the custom threat policies set in the API request body will receive a CleanResult: False response from the underlying security service, with a sub-second typical response time. This CleanResult Boolean can be used, for example, to separate clean and infected files into additional GCP buckets for eventual deletion or file quarantine/threat analysis.

Your API request can be structured using a variety of common programming languages, and in this article, I’ll provide Go code examples for your convenience.

To structure your request, use the below ready-to-run code examples and input the relevant GCP bucket information in the labeled headers. To authorize your request, copy a free-tier API key (these allow 800 API calls per month and no additional commitment) into the API Key header.

Go
 
package main

import (
     "fmt"
     "bytes"
     "mime/multipart"
     "os"
     "path/filepath"
     "io"
     "net/http"
     "io/ioutil"
)

func main() {

     url := "https://api.cloudmersive.com/virus/scan/cloud-storage/gcp-storage/single/advanced"
     method := "POST"

     payload := &bytes.Buffer{}
     writer := multipart.NewWriter(payload)
     file, errFile1 := os.Open("/path/to/file")
     defer file.Close()
     part1,
         errFile1 := writer.CreateFormFile("jsonCredentialFile",filepath.Base("/path/to/file"))
     _, errFile1 = io.Copy(part1, file)
     if errFile1 != nil {
          fmt.Println(errFile1)
          return
     }
     err := writer.Close()
     if err != nil {
          fmt.Println(err)
          return
     }


     client := &http.Client {
     }
     req, err := http.NewRequest(method, url, payload)

     if err != nil {
          fmt.Println(err)
          return
     }
     req.Header.Add("bucketName", "<string>")
     req.Header.Add("objectName", "<string>")
     req.Header.Add("allowExecutables", "<boolean>")
     req.Header.Add("allowInvalidFiles", "<boolean>")
     req.Header.Add("allowScripts", "<boolean>")
     req.Header.Add("allowPasswordProtectedFiles", "<boolean>")
     req.Header.Add("allowMacros", "<boolean>")
     req.Header.Add("allowXmlExternalEntities", "<boolean>")
     req.Header.Add("restrictFileTypes", "<string>")
     req.Header.Add("Content-Type", "multipart/form-data")
     req.Header.Add("Apikey", "YOUR-API-KEY-HERE")

     req.Header.Set("Content-Type", writer.FormDataContentType())
     res, err := client.Do(req)
     if err != nil {
          fmt.Println(err)
          return
     }
     defer res.Body.Close()

     body, err := ioutil.ReadAll(res.Body)
     if err != nil {
          fmt.Println(err)
          return
     }
     fmt.Println(string(body))
}


I’d recommend setting all the custom threat detection policies to “False” to get the most out of this solution – it can make a huge difference in improving your GCP bucket’s threat profile.

API Cloud storage Go (programming language) security

Opinions expressed by DZone contributors are their own.

Related

  • Securing Cloud Storage Access: Approach to Limiting Document Access Attempts
  • API Governance: Ensuring Control and Compliance in the Era of Digital Transformation
  • 10 Commandments of API-First Development
  • Secure Your API With JWT: Kong OpenID Connect

Partner Resources


Comments

ABOUT US

  • About DZone
  • Send feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: