Software Design and Architecture

"Migrate" comes from the Latin "migratio," meaning to move from one place to another. In information technology, migration entails understanding new systems' benefits, identifying current system shortfalls, planning, and transferring selected applications. Not all IT assets must be moved; migration can mean moving a part of them. This article will delve into the details of transferring IT assets to public clouds like AWS, Azure, or GCP. Many factors can influence the decision to switch to the cloud, such as expiring data center leases, the high costs of data center management, outdated hardware, software license renewals, geographical compliance needs, market growth, and the need to adjust resources to match demand quickly. Executive backing is crucial for a company to begin its cloud migration journey. This support is the cornerstone for any large-scale migration success. Leadership must unify their teams for the journey, as collaboration is essential. Attempts by isolated teams can lead to problems. Regular leadership meetings, whether weekly or bi-weekly, can overcome hurdles and keep the migration process on track. A pivotal step in achieving successful cloud migration is the formation of a Cloud Steering Committee. This team unites technical experts to forge the initial cloud adoption patterns. The optimal team structure includes infrastructure, security, application, and operations engineers alongside a lead architect, all steered by a Cloud Steering Committee leader. Together, they establish objectives for security, availability, reliability, scalability, data cleansing, and compliance. Embarking on the cloud migration journey can seem daunting when confronted with the vast array of applications and servers awaiting transition. There's no universal solution for migration; each enterprise faces unique challenges and opportunities. However, many organizations have successfully navigated their way to the cloud by exploring established frameworks and diverse migration strategies. Framework Evaluate and Discover Enterprises must establish a strong business case by aligning their objectives with an understanding of their current applications' age, architecture, and limitations. Leadership engagement, clear communication, and a defined purpose are essential to unify the organization and set feasible goals and timelines for the migration. In addition, a comprehensive portfolio analysis is critical, including discovering the applications, mapping interdependencies, and formulating migration strategies and priorities. This stage determines the complexity and business impact of applications, guiding the migration sequence. Starting with non-critical, simpler applications helps the team to understand the new platform and understand the gaps. Design and Migrate After classifying the applications identified in the discovery phase—whether they are web, mobile, or database systems—a standard blueprint must be created for each type. Each application requires careful design, migration, and validation following one of six migration strategies, which will be discussed later in this paper. An ethos of continuous improvement is advised, involving regular assessments of the blueprint to identify and rectify gaps. Modernize Migration isn't just about moving applications; it's about optimizing them. This means decommissioning outdated systems and steadily refining the operational model. Consider this as an evolving synergy among people, processes, and technology that improves incrementally throughout the migration journey. Migration Strategies Retain Some applications or segments of IT assets remain on-premises because they aren't suited for the cloud, don't deliver business value, or are not prepared for cloud migration. This could be due to dependencies on on-premises systems or data sovereignty issues. Retire Applications that no longer yield value to the business can be phased out. By retiring these systems, resources can be reallocated to more impactful business areas. Rehost Commonly known as "lift and shift," this strategy is popular among enterprises for its ability to facilitate a swift migration to the cloud. It requires minimal alterations to the code, database, or architecture, allowing for a more straightforward transition. Replatform Often termed "lift-tinker-and-shift," this process involves minor optimizations to applications for cloud efficiency, such as software updates, configuration tweaks, or the use of cloud-native services like Kubernetes-as-a-Service or Database-as-a-Service. An example includes transitioning from traditional database services to a cloud-based option like Amazon RDS. Repurchase This strategy comes into play when an application isn't suitable for cloud deployment or existing licensing agreements don't support a Bring-Your-Own-License (BYOL) model in the cloud. It involves switching to a Software-as-a-Service (SaaS) platform or working with Independent Software Vendors (ISVs). For example, replacing an on-premises customer relationship management (CRM) system with Salesforce. Refactor/Re-Architect This more intensive approach requires redesigning and rebuilding the application from scratch to fully exploit cloud-native features, significantly enhancing agility, scalability, and performance. Though it's the most costly and time-intensive strategy, it positions businesses to seamlessly integrate future cloud innovations with minimal effort. A typical scenario is transforming an application from a monolithic architecture to microservices. Conclusion While there is no one-size-fits-all solution for cloud migration, enterprises can benefit from analyzing successful migration journeys. Organizations can optimize their approach by emulating effective strategies and adapting them to their unique requirements. Taking the time to understand the new platform, learning from past missteps thoroughly, and refining processes are key steps toward meaningful outcomes. Moreover, it's strategic to prioritize migration projects based on business needs, considering factors such as complexity, revenue impact, operational criticality, and the timing of hardware upgrades. Additionally, investing in training for staff to master new tools and technologies is essential for a smooth transition to the cloud.

In fintech application mobile apps or the web, deploying new features in areas like loan applications requires careful validation. Traditional testing with real user data, especially personally identifiable information (PII), presents significant challenges. Synthetic transactions offer a solution, enabling the thorough testing of new functionalities in a secure and controlled environment without compromising sensitive data. By simulating realistic user interactions within the application, synthetic transactions enable developers and QA teams to identify potential issues in a controlled environment. Synthetic transactions help in ensuring that every aspect of a financial application functions correctly after any major updates or new features are rolled out. In this article, we delve into one of the approaches for using synthetic transactions. Synthetic Transactions for Financial Applications Key Business Entity At the heart of every financial application lies a key entity, be it a customer, user, or loan application itself. This entity is often defined by a unique identifier, serving as the cornerstone for transactions and operations within the system. The inception point of this entity, when it is first created, presents a strategic opportunity to categorize it as either synthetic or real. This categorization is critical, as it determines the nature of interactions the entity will undergo. Marking an entity as synthetic or for test purposes from the outset allows for a clear delineation between test and real data within the application's ecosystem. Subsequently, all transactions and operations conducted with this entity can be safely recognized as part of synthetic transactions. This approach ensures that the application's functionality can be thoroughly tested in a realistic environment. Intercepting and Managing Synthetic Transactions A critical component of implementing synthetic transactions lies in the interception and management of these transactions at the HTTP request level. Utilizing Spring's HTTP Interceptor mechanism, we can discern and process synthetic transactions by examining specific HTTP headers. The below visual outlines the coordination between a synthetic HTTP interceptor and a state manager in managing the execution of an HTTP request: Figure 1: Synthetic HTTP interceptor and state manager The SyntheticTransactionInterceptor acts as the primary gatekeeper, ensuring that only transactions identified as synthetic are allowed through the testing pathways. Below is the implementation: Java @Component public class SyntheticTransactionInterceptor implements HandlerInterceptor { protected final Logger logger = LoggerFactory.getLogger(this.getClass()); @Autowired SyntheticTransactionService syntheticTransactionService; @Autowired SyntheticTransactionStateManager syntheticTransactionStateManager; @Override public boolean preHandle(HttpServletRequest request,HttpServletResponse response, Object object) throws Exception { String syntheticTransactionId = request.getHeader("x-synthetic-transaction-uuid"); if (syntheticTransactionId != null && !syntheticTransactionId.isEmpty()){ if (this.syntheticTransactionService.validateTransactionId(syntheticTransactionId)){ logger.info(String.format("Request initiated for synthetic transaction with transaction id:%s", syntheticTransactionId)); this.syntheticTransactionStateManager.setSyntheticTransaction(true); this.syntheticTransactionStateManager.setTransactionId(syntheticTransactionId); } } return true; } } In this implementation, the interceptor looks for a specific HTTP header (x-synthetic-transaction-uuid) carrying a UUID. This UUID is not just any identifier but a validated, time-limited key designated for synthetic transactions. The validation process includes checks on the UUID's validity, its lifespan, and whether it has been previously used, ensuring a level of security and integrity for the synthetic testing process. After a synthetic ID is validated by the SyntheticTransactionInterceptor, the SyntheticTransactionStateManager plays a pivotal role in maintaining the synthetic context for the current request. The SyntheticTransactionStateManager is designed with request scope in mind, meaning its lifecycle is tied to the individual HTTP request. This scoping is essential for preserving the integrity and isolation of synthetic transactions within the application's broader operational context. By tying the state manager to the request scope, the application ensures that synthetic transaction states do not bleed over into unrelated operations or requests. Below is the implementation of the synthetic state manager: Java @Component @RequestScope public class SyntheticTransactionStateManager { private String transactionId; private boolean syntheticTransaction; public String getTransactionId() { return transactionId; } public void setTransactionId(String transactionId) { this.transactionId = transactionId; } public boolean isSyntheticTransaction() { return syntheticTransaction; } public void setSyntheticTransaction(boolean syntheticTransaction) { this.syntheticTransaction = syntheticTransaction; } } When we persist the key entity, be it a customer, user, or loan application—the application's service layer or repository layer consults the SyntheticTransactionStateManager to confirm the transaction's synthetic nature. If the transaction is indeed synthetic, the application proceeds to persist not only the synthetic identifier but also an indicator that the entity itself is synthetic. This sets the foundations for the synthetic transaction flow. This approach ensures that from the moment an entity is marked as synthetic, all related operations and future APIs, whether they involve data processing or business logic execution, are conducted in a controlled manner. For further API calls initiated from the financial application, upon reaching the microservice, we load the application context for that specific request based on the token or entity identifier provided. During the context loading, we ascertain whether the key business entity (e.g., loan application, user/customer) is synthetic. If affirmative, we then set the state manager's syntheticTransaction flag to true and also assign the synthetic transactionId from the application context. This approach negates the need to pass a synthetic transaction ID header for subsequent calls within the application flow. We only need to send a synthetic transaction ID during the initial API call that creates the key business entity. Since this step involves using explicit headers that may not be supported by the financial application, whether it's a mobile or web platform, we can manually make this first API call with Postman or a similar tool. Afterwards, the application can continue with the rest of the flow in the financial application itself. Beyond managing synthetic transactions within the application, it's also crucial to consider how external third-party API calls behave within the context of the synthetic transaction. External Third-Party API Interactions In financial applications handling key entities with personally identifiable information (PII), we conduct validations and fraud checks on user-provided data, often leveraging external third-party services. These services are crucial for tasks such as PII validation and credit bureau report retrieval. However, when dealing with synthetic transactions, we cannot make calls to these third-party services. The solution involves creating mock responses or utilizing stubs for these external services during synthetic transactions. This approach ensures that while synthetic transactions undergo the same processing logic as real transactions, they do so without the need for actual data submission to third-party services. Instead, we simulate the responses that these services would provide if they were called with real data. This allows us to thoroughly test the integration points and data-handling logic of our application. Below is the code snippet for pulling the bureau report. This call happens as part of the API call where the key entity is been created, and then subsequently we pull the applicant's bureau report: Java @Override @Retry(name = "BUREAU_PULL", fallbackMethod = "getBureauReport_Fallback") public CreditBureauReport getBureauReport(SoftPullParams softPullParams, ErrorsI error) { CreditBureauReport result = null; try { Date dt = new Date(); logger.info("UWServiceImpl::getBureauReport method call at :" + dt.toString()); CreditBureauReportRequest request = this.getCreditBureauReportRequest(softPullParams); RestTemplate restTemplate = this.externalApiRestTemplateFactory.getRestTemplate(softPullParams.getUserLoanAccountId(), "BUREAU_PULL", softPullParams.getAccessToken(), "BUREAU_PULL", error); HttpHeaders headers = this.getHttpHeaders(softPullParams); HttpEntity<CreditBureauReportRequest> entity = new HttpEntity<>(request, headers); long startTime = System.currentTimeMillis(); String uwServiceEndPoint = "/transaction"; String bureauCallUrl = String.format("%s%s", appConfig.getUnderwritingTransactionApiPrefix(), uwServiceEndPoint); if (syntheticTransactionStateManager.isSyntheticTransaction()) { result = this.syntheticTransactionService.getPayLoad(syntheticTransactionStateManager.getTransactionId(), "BUREAU_PULL", CreditBureauReportResponse.class); result.setCustomerId(softPullParams.getUserAccountId()); result.setLoanAccountId(softPullParams.getUserLoanAccountId()); } else { ResponseEntity<CreditBureauReportResponse> responseEntity = restTemplate.exchange(bureauCallUrl, HttpMethod.POST, entity, CreditBureauReportResponse.class); result = responseEntity.getBody(); } long endTime = System.currentTimeMillis(); long timeDifference = endTime - startTime; logger.info("Time taken for API call BUREAU_PULL/getBureauReport call 1: " + timeDifference); } catch (HttpClientErrorException exception) { logger.error("HttpClientErrorException occurred while calling BUREAU_PULL API, response string: " + exception.getResponseBodyAsString()); throw exception; } catch (HttpStatusCodeException exception) { logger.error("HttpStatusCodeException occurred while calling BUREAU_PULL API, response string: " + exception.getResponseBodyAsString()); throw exception; } catch (Exception ex) { logger.error("Error occurred in getBureauReport. Detail error:", ex); throw ex; } return result; } The code snippet above is quite elaborate, but we don't need to get into the details of that. What we need to focus on is the code snippet below: Java if (syntheticTransactionStateManager.isSyntheticTransaction()) { result = this.syntheticTransactionService.getPayLoad(syntheticTransactionStateManager.getTransactionId(), "BUREAU_PULL", CreditBureauReportResponse.class); result.setCustomerId(softPullParams.getUserAccountId()); result.setLoanAccountId(softPullParams.getUserLoanAccountId()); } else { ResponseEntity<CreditBureauReportResponse> responseEntity = restTemplate.exchange(bureauCallUrl, HttpMethod.POST, entity, CreditBureauReportResponse.class); result = responseEntity.getBody(); } It checks for the synthetic transaction with the SyntheticTransactionStateManager. If true, then instead of going to a third party, it calls the internal service SyntheticTransactionService to get the Synthetic Bureau report data. Synthetic Data Service Synthetic data service SyntheticTransactionServiceImpl is a general utility service whose responsibility is to pull the synthetic data from the data store, parse it, and convert it to the object type that is been passed as part of the parameter. Below is the implementation for the service: Java @Service @Qualifier("syntheticTransactionServiceImpl") public class SyntheticTransactionServiceImpl implements SyntheticTransactionService { private final Logger logger = LoggerFactory.getLogger(this.getClass()); @Autowired SyntheticTransactionRepository syntheticTransactionRepository; @Override public <T> T getPayLoad(String transactionUuid, String extPartnerServiceType, Class<T> responseType) { T payload = null; try { SyntheticTransactionPayload syntheticTransactionPayload = this.syntheticTransactionRepository.getSyntheticTransactionPayload(transactionUuid, extPartnerServiceType); if (syntheticTransactionPayload != null && syntheticTransactionPayload.getPayload() != null){ ObjectMapper objectMapper = new ObjectMapper() .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); payload = objectMapper.readValue(syntheticTransactionPayload.getPayload(), responseType); } } catch (Exception ex){ logger.error("An error occurred while getting the synthetic transaction payload, detail error:", ex); } return payload; } @Override public boolean validateTransactionId(String transactionId) { boolean result = false; try{ if (transactionId != null && !transactionId.isEmpty()) { if (UUID.fromString(transactionId).toString().equalsIgnoreCase(transactionId)) { //Removed other validation checks, this could be financial application specific check. } } } catch (Exception ex){ logger.error("SyntheticTransactionServiceImpl::validateTransactionId - An error occurred while validating the synthetic transaction id, detail error:", ex); } return result; } With the generic method getPayLoad(), we provide a high degree of reusability, capable of returning various types of synthetic responses. This reduces the need for multiple, specific mock services for different external interactions. For storing the different payloads for different types of external third-party services, we use a generic table structure as below: MySQL CREATE TABLE synthetic_transaction ( id int NOT NULL AUTO_INCREMENT, transaction_uuid varchar(36) ext_partner_service varchar(30) payload mediumtext create_date datetime DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (id) ); ext_partner_service: This is an external service identifier for which we pull the payload from the table. In this above example for bureau report, it would be BUREAU_PULL. Conclusion In our exploration of synthetic transactions within fintech applications, we've highlighted their role in enhancing the reliability, and integrity of fintech solutions. By leveraging synthetic transactions, we simulate realistic user interactions while circumventing the risks tied to handling real personally identifiable information (PII). This approach enables our developers and QA teams to rigorously test new functionalities and updates in a secure, controlled environment. Moreover, our strategy in integrating synthetic transactions through mechanisms such as HTTP interceptors and state managers showcases a versatile approach applicable across a wide array of applications. This method not only simplifies the incorporation of synthetic transactions but also significantly boosts reusability, alleviating the need to devise unique workflows for each third-party service interaction. This approach significantly enhances the reliability and security of financial application solutions, ensuring that new features can be deployed with confidence.

By Amol Gote CORE

Think of data pipeline orchestration as the backstage crew of a theater, ensuring every scene flows seamlessly into the next. In the data world, tools like Apache Airflow and AWS Step Functions are the unsung heroes that keep the show running smoothly, especially when you're working with dbt (data build tool) to whip your data into shape and ensure that the right data is available at the right time. Both tools are often used alongside dbt (data build tool), which has emerged as a powerful tool for transforming data in a warehouse. In this article, we will introduce dbt, Apache Airflow, and AWS Step Functions and then delve into the pros and cons of using Apache Airflow and AWS Step Functions for data pipeline orchestration involving dbt. A note that dbt has a paid version of dbt cloud and a free open source version; we are focussing on dbt-core, the free version of dbt. dbt (Data Build Tool) dbt-core is an open-source command-line tool that enables data analysts and engineers to transform data in their warehouses more effectively. It allows users to write modular SQL queries, which it then runs on top of the data warehouse in the appropriate order with respect to their dependencies. Key Features Version control: It integrates with Git to help track changes, collaborate, and deploy code. Documentation: Autogenerated documentation and a searchable data catalog are created based on the dbt project. Modularity: Reusable SQL models can be referenced and combined to build complex transformations. Airflow vs. AWS Step Functions for dbt Orchestration Apache Airflow Apache Airflow is an open-source tool that helps to create, schedule, and monitor workflows. It is used by data engineers/ analysts to manage complex data pipelines. Key Features Extensibility: Custom operators, executors, and hooks can be written to extend Airflow’s functionality. Scalability: Offers dynamic pipeline generation and can scale to handle multiple data pipeline workflows. Example: DAG Shell from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta default_args = { 'owner': 'airflow', 'depends_on_past': False, 'start_date': datetime.now() - timedelta(days=1), 'email_on_failure': False, 'email_on_retry': False, 'retries': 1, 'retry_delay': timedelta(minutes=5), } dag = DAG('dbt_daily_job', default_args=default_args, description='A simple DAG to run dbt jobs', schedule_interval=timedelta(days=1)) dbt_run = BashOperator( task_id='dbt_run', bash_command='dbt build --s sales.sql', dag=dag, ) slack_notify = SlackAPIPostOperator( task_id='slack_notify', dag=dag, # Replace with your actual Slack notification code ) dbt_run >> slack_notify Pros Flexibility: Apache Airflow offers unparalleled flexibility with the ability to define custom operators and is not limited to AWS resources. Community support: A vibrant open-source community actively contributes plugins and operators that provide extended functionalities. Complex workflows: Better suited to complex task dependencies and can manage task orchestration across various systems. Cons Operational overhead: Requires management of underlying infrastructure unless managed services like Astronomer or Google Cloud Composer are used. Learning curve: The rich feature set comes with a complexity that may present a steeper learning curve for some users. AWS Step Functions AWS Step Functions is a fully managed service provided by Amazon Web Services that makes it easier to orchestrate microservices, serverless applications, and complex workflows. It uses a state machine model to define and execute workflows, which can consist of various AWS services like Lambda, ECS, Sagemaker, and more. Key Features Serverless operation: No need to manage infrastructure as AWS provides a managed service. Integration with AWS Services: Seamless connection to AWS services is supported for complex orchestration. Example: State Machine Cloud Formation Template (Step Function) Shell AWSTemplateFormatVersion: '2010-09-09' Description: State Machine to run a dbt job Resources: DbtStateMachine: Type: 'AWS::StepFunctions::StateMachine' Properties: StateMachineName: DbtStateMachine RoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/service-role/StepFunctions-ECSTaskRole' DefinitionString: !Sub | Comment: "A Step Functions state machine that executes a dbt job using an ECS task." StartAt: RunDbtJob States: RunDbtJob: Type: Task Resource: "arn:aws:states:::ecs:runTask.sync" Parameters: Cluster: "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:cluster/MyECSCluster" TaskDefinition: "arn:aws:ecs:${AWS::Region}:${AWS::AccountId}:task-definition/MyDbtTaskDefinition" LaunchType: FARGATE NetworkConfiguration: AwsvpcConfiguration: Subnets: - "subnet-0193156582abfef1" - "subnet-abcjkl0890456789" AssignPublicIp: "ENABLED" End: true Outputs: StateMachineArn: Description: The ARN of the dbt state machine Value: !Ref DbtStateMachine When using AWS ECS with AWS Fargate to run dbt workflows, while you can define the dbt command in DbtTaskdefinition, it's also common to create a Docker image that contains not only the dbt environment but also the specific dbt commands you wish to run. Pros Fully managed service: AWS manages the scaling and operation under the hood, leading to reduced operational burden. AWS integration: Natural fit for AWS-centric environments, allowing easy integration of various AWS services. Reliability: Step Functions provide a high level of reliability and support, backed by AWS SLA. Cons Cost: Pricing might be higher for high-volume workflows compared to running your self-hosted or cloud-provider-managed Airflow instance. Step functions incur costs based on the number of state transitions. Locked-in with AWS: Tightly coupled with AWS services, which can be a downside if you're aiming for a cloud-agnostic architecture. Complexity in handling large workflows: While capable, it can become difficult to manage larger, more complex workflows compared to using Airflow's DAGs. There are limitations on the number of parallel executions of a State Machine. Learning curve: The service also presents a learning curve with specific paradigms, such as the Amazon States Language. Scheduling: AWS Step functions need to rely on other AWS services like AWS Eventbridge for scheduling. Summary Choosing the right tool for orchestrating dbt workflows comes down to assessing specific features and how they align with a team's needs. The main attributes that inform this decision include customization, cloud alignment, infrastructure flexibility, managed services, and cost considerations. Customization and Extensibility Apache Airflow is highly customizable and extends well, allowing teams to create tailored operators and workflows for complex requirements. Integration With AWS AWS Step Functions is the clear winner for teams operating solely within AWS, offering deep integration with the broader AWS ecosystem. Infrastructure Flexibility Apache Airflow supports a wide array of environments, making it ideal for multi-cloud or on-premises deployments. Managed Services Here, it’s a tie. For managed services, teams can opt for Amazon Managed Workflows for Apache Airflow (MWAA) for an AWS-centric approach or a vendor like Astronomer for hosting Airflow in different environments. There are also platforms like Dagster that offer similar features to Airflow and can be managed as well. This category is highly competitive and will be based on the level of integration and vendor preference. Cost at Scale Apache Airflow may prove more cost-effective for scale, given its open-source nature and the potential for optimized cloud or on-premises deployment. AWS Step Functions may be more economical at smaller scales or for teams with existing AWS infrastructure. Conclusion The choice between Apache Airflow and AWS Step Functions for orchestrating dbt workflows is nuanced. For operations deeply rooted in AWS with a preference for serverless execution and minimal maintenance, AWS Step Functions is the recommended choice. For those requiring robust customizability, diverse infrastructure support, or cost-effective scalability, Apache Airflow—whether self-managed or via a platform like Astronomer or MWAA (AWS-managed)—emerges as the optimal solution.

Welcome to the final step in creating your Collectibles portal! (for part 1, see here). In this part, we'll focus on building the front end — the last piece of the puzzle. Here's what we'll achieve: Connect the Flow Wallet. Initialize your account and mint your NFT. Check the NFT ID in your Collection. View the NFT with the NFT ID you have in your Collection. We will be using Next.js to build the front end. Let's get started! 1. Installation Setting Up Open your project flow-collectible-portal directory. Then, runnpx create-next-app@latest frontend in the terminal and press enter . This will provide you with several options. In this tutorial, we will not use Typescript, ESLint, or TailwindCSS, and we will use the src directory and the App router at the time of this article. Now you have a fresh web app ready. This is how your frontend folder looks: 2. Configuration To interact with the Flow blockchain, we will use the Flow Client Library (FCL) to manage wallet connections, run scripts, and send transactions in our application. It will allow us to write complete Cadence functions and run them as Javascript functions. To get started, let's install FCL for our app by running the following command: Shell npm install @onflow/fcl --save After installing FCL, we need to configure it. Here's what you need to do: Inside the app folder create a new folder named flow and add a file named config.js. In this file, set up the configuration for the FCL, such as specifying the Access Node and wallet discovery endpoint. This helps you to choose between using a testnet or a local emulator. You will also want to specify the Collectibles contract address we deployed in Part 1. Add the following code to the config.js file: JavaScript import { config } from "@onflow/fcl"; config({ "app.detail.title": "Flow Name Service", "app.detail.icon": "https://placekitten.com/g/200/200", "accessNode.api": "https://rest-testnet.onflow.org", "discovery.wallet": "https://fcl-discovery.onflow.org/testnet/authn", "0xCollectibles": "ADD YOUR CONTRACT ACCOUNT ADDRESS", "0xNonFungibleToken": "0x631e88ae7f1d7c20", }); Now, you're all set up to use the FCL in your app. 3. Authentication To verify a user's identity in an app, you can use several functions: For logging in, call fcl.logIn(). For signing up, call fcl.signUp(). For logging out, call fcl.unauthenticate(). Let’s learn how we can implement these fcl functions in your front end. First, we will add the following code to our page.js file inside the app directory. This will import some dependencies, set up some initial useState for parts of our app, and build a basic UI. To make sure it looks nice, delete the page.module.css file inside the app directory and instead make a file called page.css. Then, paste the contents of this file inside of it. Now, we can write out our initial page. JavaScript "use client"; import React, { useState, useEffect, useRef } from "react"; import * as fcl from "@onflow/fcl"; import "./page.css"; import "./flow/config"; export default function Page() { const [currentUser, setCurrentUser] = useState({ loggedIn: false, addr: undefined, }); const urlInputRef = useRef(); const nameInputRef = useRef(); const idInputRef = useRef(); const [isInitialized, setIsInitialized] = useState(); const [collectiblesList, setCollectiblesList] = useState([]); const [loading, setLoading] = useState(false); const [ids, setIds] = useState([]); const [nft, setNFT] = useState({}); useEffect(() => fcl.currentUser.subscribe(setCurrentUser), []); function handleInputChange(event) { const inputValue = event.target.value; if (/^\d+$/.test(inputValue)) { idInputRef.current = +inputValue; } else { console.error("Invalid input. Please enter a valid integer."); } } return ( <div> <div className="navbar"> <h1>Flow Collectibles Portal</h1> <span>Address: {currentUser?.addr ?? "NO Address"}</span> <button onClick={currentUser.addr ? fcl.unauthenticate : fcl.logIn}> {currentUser.addr ? "Log Out" : "Connect Wallet"} </button> </div> {currentUser.loggedIn ? ( <div className="main"> <div className="mutate"> <h1>Mutate Flow Blockchain</h1> <form onSubmit={(event) => { event.preventDefault(); } > <input type="text" placeholder="enter name of the NFT" ref={nameInputRef} /> <input type="text" placeholder="enter a url" ref={urlInputRef} /> <button type="submit">Mint</button> </form> <mark>Your Collection will be initialized while minting NFT.</mark> </div> <div className="query"> <h1>Query Flow Blockchain</h1> <mark>Click below button to check</mark> <button>Check Collection</button> <p> Is your collection initialized: {isInitialized ? "Yes" : "No"} </p> <button onClick={viewIds}> View NFT IDs you hold in your collection </button> <p>NFT Id: </p> </div> <div className="view"> <h1>View Your NFT</h1> <input type="text" placeholder="enter your NFT ID" onChange={handleInputChange} /> <button>View NFT</button> <div className="nft-card"> <p>NFT id: </p> <p>NFT name: </p> <img src="" alt="" /> </div> </div> </div> ) : ( <div className="main-2"> <h1>Connect Wallet to mint NFT!!</h1> </div> )} </div> ); } After adding this code, run npm run dev to make sure everything loads correctly. 4. Querying the Flow Blockchain Before getting a deep dive into how we can use fcl to query the Flow blockchain, add these Cadence script codes after the handleInput function in the page.js file. JavaScript const CHECK_COLLECTION = ` import NonFungibleToken from 0xNonFungibleToken import Collectibles from 0xCollectibles pub fun main(address: Address): Bool? { return Collectibles.checkCollection(_addr: address) }` const GET_NFT_ID = ` import NonFungibleToken from 0xNonFungibleToken import Collectibles from 0xCollectibles pub fun main(user: Address): [UInt64] { let collectionCap = getAccount(user).capabilities.get <&{Collectibles.CollectionPublic}>(/public/NFTCollection) ?? panic("This public capability does not exist.") let collectionRef = collectionCap.borrow()! return collectionRef.getIDs() } ` const GET_NFT = ` import NonFungibleToken from 0xNonFungibleToken import Collectibles from 0xCollectibles pub fun main(user: Address, id: UInt64): &NonFungibleToken.NFT? { let collectionCap= getAccount(user).capabilities.get<&{Collectibles.CollectionPublic}>(/public/NFTCollection) ?? panic("This public capability does not exist.") let collectionRef = collectionCap.borrow()! return collectionRef.borrowNFT(id: id) } With our Cadence scripts ready to go, we can now declare some JavaScript functions and pass in the Cadence constants into the `fcl` queries. JavaScript async function checkCollectionInit() { const isInit = await fcl.query({ cadence: CHECK_COLLECTION, args: (arg,t) => [arg(currentUser?.addr, t.Address)], }); console.log(isInit); } async function viewNFT() { console.log(idInputRef.current); const nfts = await fcl.query({ cadence: GET_NFT, args: (arg,t) => [arg(currentUser?.addr,t.Address), arg(idInputRef.current, t.UInt64)] }); setNFT(nfts); console.log(nfts); } async function viewIds() { const ids = await fcl.query({ cadence: GET_NFT_ID, args: (arg,t) => [arg(currentUser?.addr,t.Address)] }); setIds(ids); console.log(ids); } Now, let’s take a look at all the functions we’ve written. There are two things to notice: The fcl.query And the args: (arg,t) => [arg(addr,t.Address)], line. Since scripts are similar to view functions in Solidity and don't require any gas fees to run, we are essentially just querying the blockchain. So, we use fcl.query to run scripts on Flow. In order to query something, we need to pass an argument. For that, we use arg, which is a function that takes a string value representing the argument and t, which is an object that contains all the different data types that Cadence has. So we can tell arg how to encode and decode the argument we are passing. 5. Mutating the Flow Blockchain While our previous functions were just “read-only,” our next ones will have actions that can mutate the blockchain state and write to it, a.k.a. “mint an NFT.” To do this, we’ll write another Cadence script as a constant. JavaScript const MINT_NFT = ` import NonFungibleToken from 0xNonFungibleToken import Collectibles from 0xCollectibles transaction(name:String, image:String){ let receiverCollectionRef: &{NonFungibleToken.CollectionPublic} prepare(signer:AuthAccount){ // initialise account if signer.borrow<&Collectibles.Collection>(from: Collectibles.CollectionStoragePath) == nil { let collection <- Collectibles.createEmptyCollection() signer.save(<-collection, to: Collectibles.CollectionStoragePath) let cap = signer.capabilities.storage.issue<&{Collectibles.CollectionPublic}>(Collectibles.CollectionStoragePath) signer.capabilities.publish( cap, at: Collectibles.CollectionPublicPath) } //takes the receiver collection refrence self.receiverCollectionRef = signer.borrow<&Collectibles.Collection>(from: Collectibles.CollectionStoragePath) ?? panic("could not borrow Collection reference") } execute{ let nft <- Collectibles.mintNFT(name:name, image:image) self.receiverCollectionRef.deposit(token: <-nft) } } Now add the below function after the transaction code to the page.js file. JavaScript async function mint() { try{ const txnId = await fcl.mutate({ cadence: MINT_NFT, args: (arg,t) => [arg(name,t.String), arg(image, t.String)], payer: fcl.authz, proposer: fcl.authz, authorizations: [fcl.authz], limit:999,}); } catch(error){ console.error('Minting failed:' error) } console.log(txnId); } As for the function, the fcl.mutate syntax is the same as fcl.query. However, we do provide several extra parameters, such as the following: YAML payer: fcl.authz, proposer: fcl.authz, authorizations: [fcl.authz], limit: 50, These are Flow-specific things that define which account will be paying for the transaction (payer), broadcasting the transaction (proposer), and the accounts from which we need authorizations. (In case an account has multiple keys attached, it can behave like a multi-sig wallet.) fcl.authz refers to the currently connected account. limit is like gasLimit in the Ethereum world, which places an upper limit on the maximum amount of computation. If the computation crosses the limit, then the transaction will fail. We’ll need to add one more function that will call and handle the mintNFT function we just made. JavaScript const saveCollectible = async () => { if (urlInputRef.current.value.length > 0 && nameInputRef.current.value.length > 0) { try { setLoading(true); const transaction = await mintNFT(nameInputRef.current.value, urlInputRef.current.value); console.log('transactionID:', transaction); // Handle minting success (if needed) } catch (error) { console.error('Minting failed:', error); // Handle minting failure (if needed) } finally { setLoading(false); } } else { console.log('Empty input. Try again.'); } }; 6. Final Code With our main functions in place, we can now plug them into our UI. However, before we do that, we’ll add some useEffect calls to help load the initial state. You can add these right above the already existing useEffect call. JavaScript useEffect(() => { checkCollectionInit(); viewNFT(); }, [currentUser]); useEffect(() => { if (currentUser.loggedIn) { setCollectiblesList(collectiblesList); console.log('Setting collectibles...'); } }, [currentUser]); Now, back in our return section with the UI, we can add our functions to the appropriate parts of the app. JavaScript return ( <div> <div className="navbar"> <h1>Flow Collectibles Portal</h1> <span>Address: {currentUser?.addr ?? "NO Address"}</span> <button onClick={currentUser.addr ? fcl.unauthenticate : fcl.logIn}> {currentUser.addr ? "Log Out" : "Connect Wallet"} </button> </div> {currentUser.loggedIn ? ( <div className="main"> <div className="mutate"> <h1>Mutate Flow Blockchain</h1> <form onSubmit={(event) => { event.preventDefault(); saveCollectible(); } > <input type="text" placeholder="enter name of the NFT" ref={nameInputRef} /> <input type="text" placeholder="enter a url" ref={urlInputRef} /> <button type="submit">Mint</button> </form> <mark>Your Collection will be initialized while minting NFT.</mark> </div> <div className="query"> <h1>Query Flow Blockchain</h1> <mark>Click below button to check</mark> <button onClick={checkCollectionInit}>Check Collection</button> <p> Is your collection initialized: {isInitialized ? "Yes" : "No"} </p> <button onClick={viewIds}> View NFT IDs you hold in your collection </button> <p>NFT Id: </p> {ids.map((id) => ( <p key={id}>{id}</p> ))} </div> <div className="view"> <h1>View Your NFT</h1> <input type="text" placeholder="enter your NFT ID" onChange={handleInputChange} /> <button onClick={viewNFT}>View NFT</button> <div className="nft-card"> <p>NFT id: {nft.id}</p> <p>NFT name: {nft.name}</p> <img src={nft.image} alt={nft.name} /> </div> </div> </div> ) : ( <div className="main-2"> <h1>Connect Wallet to mint NFT!!</h1> </div> )} </div> ); Check the final code here. Now that the app is complete, let’s walk through how to use it! First, connect your wallet by clicking the “Connect Wallet” button in the top right. Now, you can mint an NFT! Enter the name of your NFT and paste in a link to the image you want to use. After you click “mint,” it will prompt you to sign a transaction with your wallet. It might take a little while for the transaction to complete. After it completes, you should be able to click the bottom button to view the IDs of your NFTs. If this is your first one, then the ID should be just “1”. Now, you can copy the ID of your NFT, paste it into the View section, and click “View NFT.” Conclusion Well done! You've finished part 2 of the Collectibles portal project. In summary, we focused on building the front end of our Collectibles portal. We did this by: Creating an app with Next.js Connecting the Flow Wallet Creating our very own NFTs for minting Viewing your NFT Have a really great day!

By John Vester CORE

I started research for an article on how to add a honeytrap to a GitHub repo. The idea behind a honeypot weakness is that a hacker will follow through on it and make his/her presence known in the process. My plan was to place a GitHub personal access token in an Ansible vault protected by a weak password. Should an attacker crack the password and use the token to clone the private repository, a webhook should have triggered and mailed a notification that the honeypot repo has been cloned and the password cracked. Unfortunately, GitHub seems not to allow webhooks to be triggered after cloning, as is the case for some of its higher-level actions. This set me thinking that platforms as standalone systems are not designed with Dev(Sec)Ops integration in mind. DevOps engineers have to bite the bullet and always find ways to secure pipelines end-to-end. I, therefore, instead decided to investigate how to prevent code theft using tokens or private keys gained by nefarious means. Prevention Is Better Than Detection It is not best practice to have secret material on hard drives thinking that root-only access is sufficient security. Any system administrator or hacker that is elevated to root can view the secret in the open. They should, rather, be kept inside Hardware Security Modules (HSMs) or a secret manager, at the very least. Furthermore, tokens and private keys should never be passed in as command line arguments since they might be written to a log file. A way to solve this problem is to make use of a super-secret master key to initiate proceedings and finalize using short-lived lesser keys. This is similar to the problem of sharing the first key in applied cryptography. Once the first key has been agreed upon, successive transactions can be secured using session keys. It goes beyond saying that the first key has to be stored in Hardware Security Modules, and all operations against it have to happen inside an HSM. I decided to try out something similar when Ansible clones private Git repositories. Although I will illustrate at the hand of GitHub, I am pretty sure something similar can be set up for other Git platforms as well. First Key GitHub personal access tokens can be used to perform a wide range of actions on your GitHub account and its repositories. It authenticates and authorizes from both the command line and the GitHub API. It clearly can serve as the first key. Personal access tokens are created by clicking your avatar in the top right and selecting Settings: A left nav panel should appear from where you select Developer settings: The menu for personal access tokens will display where you can create the token: I created a classic token and gave it the following scopes/permissions: repo, admin:public_key, user, and admin:gpg_key. Take care to store the token in a reputable secret manager from where it can be copied and pasted when the Ansible play asks for it when it starts. This secret manager should clear the copy buffer after a few seconds to prevent attacks utilizing attention diversion. vars_prompt: - name: github_token prompt: "Enter your github personal access token?" private: true Establishing the Session GitHub deployment keys give access to private repositories. They can be created by an API call or from the repo's top menu by clicking on Settings: With the personal access token as the first key, a deployment key can finish the operation as the session key. Specifically, Ansible authenticates itself using the token, creates the deployment key, authorizes the clone, and deletes it immediately afterward. The code from my previous post relied on adding Git URLs that contain the tokens to the Ansible vault. This has now been improved to use temporary keys as envisioned in this post. An Ansible role provided by Asif Mahmud has been amended for this as can be seen in the usual GitHub repo. The critical snippets are: - name: Add SSH public key to GitHub account ansible.builtin.uri: url: "https://api.{{ git_server_fqdn }/repos/{{ github_account_id }/{{ repo }/keys" validate_certs: yes method: POST force_basic_auth: true body: title: "{{ key_title }" key: "{{ key_content.stdout }" read_only: true body_format: json headers: Accept: application/vnd.github+json X-GitHub-Api-Version: 2022-11-28 Authorization: "Bearer {{ github_access_token }" status_code: - 201 - 422 register: create_result The GitHub API is used to add the deploy key to the private repository. Note the use of the access token typed in at the start of play to authenticate and authorize the request. - name: Clone the repository shell: | GIT_SSH_COMMAND="ssh -i {{ key_path } -v -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" {{ git_executable } clone git@{{ git_server_fqdn }:{{ github_account_id }/{{ repo }.git {{ clone_dest } - name: Switch branch shell: "{{ git_executable } checkout {{ branch }" args: chdir: "{{ clone_dest }" The repo is cloned, followed by a switch to the required branch. - name: Delete SSH public key ansible.builtin.uri: url: "https://api.{{ git_server_fqdn }/repos/{{ github_account_id }/{{ repo }/keys/{{ create_result.json.id }" validate_certs: yes method: DELETE force_basic_auth: true headers: Accept: application/vnd.github+json X-GitHub-Api-Version: 2022-11-28 Authorization: "Bearer {{ github_access_token }" status_code: - 204 Deletion of the deployment key happens directly after the clone and switch, again via the API. Conclusion The short life of the deployment key enhances the security of the DevOps pipeline tremendously. Only the token has to be kept secured at all times as is the case for any first key. Ideally, you should integrate Ansible with a compatible HSM platform. I thank Asif Mahmud for using their code to illustrate the concept of using temporary session keys when cloning private Git repositories.

The NIST AI RMF (National Institute of Standards and Technology Artificial Intelligence Risk Management Framework) provides a structured framework for identifying, assessing, and mitigating risks associated with artificial intelligence technologies, addressing complex challenges such as algorithmic bias, data privacy, and ethical considerations, thus helping organizations ensure the security, reliability, and ethical use of AI systems. How Do AI Risks Differ From Traditional Software Risks? AI risks differ from traditional software risks in several key ways: Complexity: AI systems often involve complex algorithms, machine learning models, and large datasets, which can introduce new and unpredictable risks. Algorithmic bias: AI systems can exhibit bias or discrimination based on factors such as the training data used to develop the models. This can result in unintended outcomes and consequences, which may not be part of traditional software systems. Opacity and lack of interpretability: AI algorithms, particularly deep learning models, can be opaque and difficult to interpret. This can make it challenging to understand how AI systems make decisions or predictions, leading to risks related to accountability, transparency, and trust. Data quality and bias: AI systems rely heavily on data, and issues such as data quality, incompleteness, and bias can significantly impact their performance and reliability. Traditional software may also rely on data, but the implications of data quality issues may be more noticeable in AI systems, affecting the accuracy, and effectiveness of AI-driven decisions. Adversarial attacks: AI systems may be vulnerable to adversarial attacks, where malicious actors manipulate inputs to deceive or manipulate the system's behavior. Adversarial attacks exploit vulnerabilities in AI algorithms and can lead to security breaches, posing distinct risks compared to traditional software security threats. Ethical and societal implications: AI technologies raise ethical and societal concerns that may not be as prevalent in traditional software systems. These concerns include issues such as privacy violations, job displacement, loss of autonomy, and reinforcement of biases. Regulatory and compliance challenges: AI technologies are subject to a rapidly evolving regulatory landscape, with new laws and regulations emerging to address AI-specific risks and challenges. Traditional software may be subject to similar regulations, but AI technologies often raise novel compliance issues related to fairness, accountability, transparency, and bias mitigation. Cost: The expense associated with managing an AI system exceeds that of regular software, as it often requires ongoing tuning to align with the latest models, training, and self-updating processes. Effectively managing AI risks requires specialized knowledge, tools, and frameworks tailored to the unique characteristics of AI technologies and their potential impact on individuals, organizations, and society as a whole. Key Considerations of the AI RMF The AI RMF refers to an AI system as an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. The AI RMF helps organizations effectively identify, assess, mitigate, and monitor risks associated with AI technologies throughout the lifecycle. It addresses various challenges, like data quality issues, model bias, adversarial attacks, algorithmic transparency, and ethical considerations. Key considerations include: Risk identification Risk assessment and prioritization Control selection and tailoring Implementation and integration Monitoring and evaluation Ethical and social implications Interdisciplinary collaboration Key Functions of the Framework Following are the essential functions within the NIST AI RMF that help organizations effectively identify, assess, mitigate, and monitor risks associated with AI technologies. Image courtesy of NIST AI RMF Playbook Govern Governance in the NIST AI RMF refers to the establishment of policies, processes, structures, and mechanisms to ensure effective oversight, accountability, and decision-making related to AI risk management. This includes defining roles and responsibilities, setting risk tolerance levels, establishing policies and procedures, and ensuring compliance with regulatory requirements and organizational objectives. Governance ensures that AI risk management activities are aligned with organizational priorities, stakeholder expectations, and ethical standards. Map Mapping in the NIST AI RMF involves identifying and categorizing AI-related risks, threats, vulnerabilities, and controls within the context of the organization's AI ecosystem. This includes mapping AI system components, interfaces, data flows, dependencies, and associated risks to understand the broader risk landscape. Mapping helps organizations visualize and prioritize AI-related risks, enabling them to develop targeted risk management strategies and allocate resources effectively. It may also involve mapping AI risks to established frameworks, standards, or regulations to ensure comprehensive coverage and compliance. Measurement Measurement in the NIST AI RMF involves assessing and quantifying AI-related risks, controls, and performance metrics to evaluate the effectiveness of risk management efforts. This includes conducting risk assessments, control evaluations, and performance monitoring activities to measure the impact of AI risks on organizational objectives and stakeholder interests. Measurement helps organizations identify areas for improvement, track progress over time, and demonstrate the effectiveness of AI risk management practices to stakeholders. It may also involve benchmarking against industry standards or best practices to identify areas for improvement and drive continuous improvement. Manage Management in the NIST AI RMF refers to the implementation of risk management strategies, controls, and mitigation measures to address identified AI-related risks effectively. This includes implementing selected controls, developing risk treatment plans, and monitoring AI systems' security posture and performance. Management activities involve coordinating cross-functional teams, communicating with stakeholders, and adapting risk management practices based on changing risk environments. Effective risk management helps organizations minimize the impact of AI risks on organizational objectives, stakeholders, and operations while maximizing the benefits of AI technologies. Key Components of the Framework The NIST AI RMF consists of two primary components: Foundational Information This part includes introductory materials, background information, and context-setting elements that provide an overview of the framework's purpose, scope, and objectives. It may include definitions, principles, and guiding principles relevant to managing risks associated with artificial intelligence (AI) technologies. Core and Profiles This part comprises the core set of processes, activities, and tasks necessary for managing AI-related risks, along with customizable profiles that organizations can tailor to their specific needs and requirements. The core provides a foundation for risk management, while profiles allow organizations to adapt the framework to their unique circumstances, addressing industry-specific challenges, regulatory requirements, and organizational priorities. Significance of AI RMF Based on Roles Benefits for Developers Guidance on risk management: The AI RMF provides developers with structured guidance on identifying, assessing, mitigating, and monitoring risks associated with AI technologies. Compliance with standards and regulations: The AI RMF helps developers ensure compliance with relevant standards, regulations, and best practices governing AI technologies. By referencing established NIST guidelines, such as NIST SP 800-53, developers can identify applicable security and privacy controls for AI systems. Enhanced security and privacy: By incorporating security and privacy controls recommended in the AI RMF, developers can mitigate the risks of data breaches, unauthorized access, and other security threats associated with AI systems. Risk awareness and mitigation: The AI RMF raises developers' awareness of potential risks and vulnerabilities inherent in AI technologies, such as data quality issues, model bias, adversarial attacks, and algorithmic transparency. Cross-disciplinary collaboration: The AI RMF emphasizes the importance of interdisciplinary collaboration between developers, cybersecurity experts, data scientists, ethicists, legal professionals, and other stakeholders in managing AI-related risks. Quality assurance and testing: The AI RMF encourages developers to incorporate risk management principles into the testing and validation processes for AI systems. Benefits for Architects Designing secure and resilient systems: Architects play a crucial role in designing the architecture of AI systems. By incorporating principles and guidelines from the AI RMF into the system architecture, architects can design AI systems that are secure, resilient, and able to effectively manage risks associated with AI technologies. This includes designing robust data pipelines, implementing secure APIs, and integrating appropriate security controls to mitigate potential vulnerabilities. Ensuring compliance and governance: Architects are responsible for ensuring that AI systems comply with relevant regulations, standards, and organizational policies. By integrating compliance requirements into the system architecture, architects can ensure that AI systems adhere to legal and ethical standards while protecting sensitive information and user privacy. Addressing ethical and societal implications: Architects need to consider the ethical and societal implications of AI technologies when designing system architectures. Architects can leverage the AI RMF to incorporate mechanisms for ethical decision-making, algorithmic transparency, and user consent into the system architecture, ensuring that AI systems are developed and deployed responsibly. Supporting continuous improvement: The AI RMF promotes a culture of continuous improvement in AI risk management practices. Architects can leverage the AI RMF to establish mechanisms for monitoring and evaluating the security posture and performance of AI systems over time. Comparison of AI Risk Frameworks Framework Strengths Weaknesses NIST AI RMF Comprehensive coverage of AI-specific risks Integration with established NIST cybersecurity guidelines Interdisciplinary approach Alignment with regulatory requirements Emphasis on continuous improvement May require customization to address specific organizational needs Focus on the US-centric regulatory landscape ISO/IEC 27090 Widely recognized international standards ISO/IEC 27090 is designed to integrate seamlessly with ISO/IEC 27001, the international standard for information security management systems (ISMS). Provides comprehensive guidance on managing risks associated with AI technologies The standard follows a structured approach, incorporating the Plan-Do-Check-Act (PDCA) cycle. Lack of specificity in certain areas, as it aims to provide general guidance applicable to a wide range of organizations and industries Implementing ISO/IEC 27090 can be complex, particularly for organizations that are new to information security management or AI risk management. The standard's comprehensive nature and technical language may require significant expertise and resources to understand and implement effectively. IEEE P7006 Focus on data protection considerations in AI systems, particularly those related to personal data Comprehensive guidelines for ensuring privacy, fairness, transparency, and accountability Limited scope to personal data protection May not cover all aspects of AI risk management Fairness, Accountability, and Transparency (FAT) Framework Emphasis on ethical dimensions of AI, including fairness, accountability, transparency, and explainability Provides guidelines for evaluating and mitigating ethical risks Not a comprehensive risk management framework May lack detailed guidance on technical security controls IBM AI Governance Framework Focus on governance aspects of AI projects Covers various aspects of AI lifecycle, including data management, model development, deployment, and monitoring Emphasis on transparency, fairness, and trustworthiness Developed by a specific vendor and may be perceived as biased May not fully address regulatory requirements beyond IBM's scope Google AI Principles Clear principles for ethical AI development and deployment Emphasis on fairness, privacy, accountability, and societal impact Provides guidance for responsible AI practices Not a comprehensive risk management framework Lacks detailed implementation guidance AI Ethics Guidelines from Industry Consortia Developed by diverse stakeholders, including industry, academia, and civil society Provides a broad perspective on ethical AI considerations Emphasis on collaboration and knowledge sharing Not comprehensive risk management frameworks May lack detailed implementation guidance Conclusion The NIST AI Risk Management Framework offers a comprehensive approach to addressing the complex challenges associated with managing risks in artificial intelligence (AI) technologies. Through its foundational information and core components, the framework provides organizations with a structured and adaptable methodology for identifying, assessing, mitigating, and monitoring risks throughout the AI lifecycle. By leveraging the principles and guidelines outlined in the framework, organizations can enhance the security, reliability, and ethical use of AI systems while ensuring compliance with regulatory requirements and stakeholder expectations. However, it's essential to recognize that effectively managing AI-related risks requires ongoing diligence, collaboration, and adaptation to evolving technological and regulatory landscapes. By embracing the NIST AI RMF as a guiding framework, organizations can navigate the complexities of AI risk management with confidence and responsibility, ultimately fostering trust and innovation in the responsible deployment of AI technologies.

By Josephine Eskaline Joyce CORE

Microservices have emerged as a transformative architectural approach in the realm of software development, offering a paradigm shift from monolithic structures to a more modular and scalable system. At its core, microservices involve breaking down complex applications into smaller, independently deployable services that communicate seamlessly, fostering agility, flexibility, and ease of maintenance. This decentralized approach allows developers to focus on specific functionalities, enabling rapid development, continuous integration, and efficient scaling to meet the demands of modern, dynamic business environments. As organizations increasingly embrace the benefits of microservices, this article explores the key principles, advantages, and challenges associated with this architectural style, shedding light on its pivotal role in shaping the future of software design and deployment. A fundamental characteristic of microservices applications is the ability to design, develop, and deploy each microservice independently, utilizing diverse technology stacks. Each microservice functions as a self-contained, autonomous application with its own dedicated persistent storage, whether it be a relational database, a NoSQL DB, or even a legacy file storage system. This autonomy enables individual microservices to scale independently, facilitating seamless real-time infrastructure adjustments and enhancing overall manageability. NCache Caching Layer in Microservice Architecture In scenarios where application transactions surge, bottlenecks may persist, especially in architectures where microservices store data in non-scalable relational databases. Simply deploying additional instances of the microservice doesn't alleviate the problem. To address these challenges, consider integrating NCache as a distributed cache at the caching layer between microservices and datastores. NCache serves not only as a cache but also functions as a scalable in-memory publisher/subscriber messaging broker, facilitating asynchronous communication between microservices. Microservice Java application performance optimization can be achieved by the cache techniques like Cache item locking, grouping Cache data, Hibernate Caching, SQL Query, data structure, spring data cache technique pub-sub messaging, and many more with NCache. Please check the out-of-the-box features provided by NCache. Using NCache as Hibernate Second Level Java Cache Hibernate First-Level Cache The Hibernate first-level cache serves as a fundamental standalone (in-proc) cache linked to the Session object, limited to the current session. Nonetheless, a drawback of the first-level cache is its inability to share objects between different sessions. If the same object is required by multiple sessions, each triggers a database trip to load it, intensifying database traffic and exacerbating scalability issues. Furthermore, when the session concludes, all cached data is lost, necessitating a fresh fetch from the database upon the next retrieval. Hibernate Second-Level Cache For high-traffic Hibernate applications relying solely on the first-level cache, deployment in a web farm introduces challenges related to cache synchronization across servers. In a web farm setup, each node operates a web server—such as Apache, Oracle WebLogic, etc.—with multiple instances of httpd processes to serve requests. Each Hibernate first-level cache in these HTTP worker processes maintains a distinct version of the same data directly cached from the database, posing synchronization issues. This is why Hibernate offers a second-level cache with a provider model. The Hibernate second-level cache enables you to integrate third-party distributed (out-proc) caching providers to cache objects across sessions and servers. Unlike the first-level cache, the second-level cache is associated with the SessionFactory object and is accessible to the entire application, extending beyond a single session. Enabling the Hibernate second-level cache results in the coexistence of two caches: the first-level cache and the second-level cache. Hibernate endeavors to retrieve objects from the first-level cache first; if unsuccessful, it attempts to fetch them from the second-level cache. If both attempts fail, the objects are directly loaded from the database and cached. This configuration substantially reduces database traffic, as a significant portion of the data is served by the second-level distributed cache. NCache Java has implemented a Hibernate second-level caching provider by extending org.hibernate.cache.CacheProvider. Integrating NCache Java Hibernate distributed caching provider with the Hibernate application requires no code changes. This integration enables you to scale your Hibernate application to multi-server configurations without the database becoming a bottleneck. NCache also delivers enterprise-level distributed caching features, including data size management, data synchronization across servers, and more. To incorporate the NCache Java Hibernate caching provider, a simple modification of your hibernate.cfg.xml and ncache.xml is all that is required. Thus, with the NCache Java Hibernate distributed cache provider, you can achieve linear scalability for your Hibernate applications seamlessly, requiring no alterations to your existing code. Code Snippet Java // Configure Hibernate properties programmatically Properties hibernateProperties = new Properties(); hibernateProperties.put("hibernate.connection.driver_class", "org.h2.Driver"); hibernateProperties.put("hibernate.connection.url", "jdbc:h2:mem:testdb"); hibernateProperties.put("hibernate.show_sql", "false"); hibernateProperties.put("hibernate.hbm2ddl.auto", "create-drop"); hibernateProperties.put("hibernate.cache.use_query_cache", "true"); hibernateProperties.put("hibernate.cache.use_second_level_cache", "true"); hibernateProperties.put("hibernate.cache.region.factory_class", "org.hibernate.cache.jcache.internal.JCacheRegionFactory"); hibernateProperties.put("hibernate.javax.cache.provider", "com.alachisoft.ncache.hibernate.jcache.HibernateNCacheCachingProvider"); // Set other Hibernate properties as needed Configuration configuration = new Configuration() .setProperties(hibernateProperties).addAnnotatedClass(Product.class); Logger.getLogger("org.hibernate").setLevel(Level.OFF); // Build the ServiceRegistry ServiceRegistry serviceRegistry = new StandardServiceRegistryBuilder() .applySettings(configuration.getProperties()).build(); // Build the SessionFactory SessionFactory factory = configuration.buildSessionFactory(serviceRegistry); // Create a List of Product objects ArrayList<Product> products = (ArrayList<Product>) getProducts(); // Open a new Hibernate session to save products to the database. This also caches it try (Session session = factory.openSession()) { Transaction transaction = session.beginTransaction(); // save() method saves products to the database and caches it too System.out.println("ProductID, Name, Price, Category"); for (Product product : products) { System.out.println("- " + product.getProductID() + ", " + product.getName() + ", " + product.getPrice() + ", " + product.getCategory()); session.save(product); } transaction.commit(); System.out.println(); // Now open a new session to fetch products from the DB. // But, these products are actually fetched from the cache try (Session session = factory.openSession()) { List<Product> productList = (List<Product>) session.createQuery("from Product").list(); if (productList != null) { printProductDetails(productList); } } Integrate NCache with Hibernate to effortlessly cache the results of queries. When these objects are subsequently fetched by Hibernate, they are retrieved from the cache, thereby avoiding a costly trip to the database. From the above code sample, the products are saved in the database, and it also caches; now, when the new session opens to fetch the product details, it will fetch from the Cache and avoid an unnecessary database trip. Learn more about Hibernate Caching Scaling With NCache Pub/Sub Messaging NCache is a distributed in-memory caching solution designed for .NET. Its compatibility extends to Java through a native client and third-party integrations, ensuring seamless support for both platforms. NCache serves as an in-memory distributed data store tailored for .NET and Java, offering a feature-rich, in-memory pub/sub mechanism for event-driven communication. This makes it straightforward to set up NCache as a messaging broker, employing the Pub/Sub model for seamless asynchronous communication between microservices. Using NCache In-Memory Pub/Sub for Microservices NCache enables Pub/Sub functionality by establishing a topic where microservices can publish and subscribe to events. These events are published to the NCache message broker outside the microservice. Within each subscribing microservice, there exists an event handler to manage the corresponding event once it has been published by the originating microservice. In the realm of Java microservices, NCache functions as an event bus or message broker, facilitating the relay of messages to one or multiple subscribers. In the context of Pub/Sub models that necessitate a communication channel, NCache serves as a medium for topics. This entails the publisher dispatching messages to the designated topic and subscribers receiving notifications through the same topic. Employing NCache as the medium for topics promotes loose coupling within the model, offering enhanced abstraction and additional advantages for distributed topics. Publish The code snippet below initializes the messageService object using NCache MessagingService package. Initializing the Topic Java // Create a Topic in NCache. MessagingService messagingService = cache.getMessagingService(); Topic topic = messagingService.createTopic(topicName); // Create a thread pool for publishers ExecutorService publisherThreadPool = Executors.newFixedThreadPool(2); The below code snippet used to define register the subscribers to this topic Register subscribers to this Topic MessageReceivedListener subscriptionListener1 = new MessageReceivedListener() { @Override public void onMessageReceived(Object o, MessageEventArgs messageEventArgs) { messageReceivedSubscription1(messageEventArgs.getMessage()); } }; MessageReceivedListener subscriptionListener2 = new MessageReceivedListener() { @Override public void onMessageReceived(Object o, MessageEventArgs messageEventArgs) { messageReceivedSubscription2(messageEventArgs.getMessage()); } }; TopicSubscription subscription1 = topic.createSubscription(subscriptionListener1); TopicSubscription subscription2 = topic.createSubscription(subscriptionListener2); NCache provides two variants of durable subscriptions to cater to the message durability needs within your Java microservices: Shared Durable Subscriptions: This allows multiple subscribers to connect to a single subscription. The Round Robin approach is employed to distribute messages among the various subscribers. Even if a subscriber exits the network, messages persistently flow between the active subscribers. Exclusive Durable Subscriptions: In this type, only one active subscriber is allowed on a subscription at any given time. No new subscriber requests are accepted for the same subscription until the existing connection is active. Learn more Pub/Sub Messaging with NCache implementation here Pub/Sub Messaging in Cache: An Overview SQL Query on Cache NCache provides your microservices with the capability to perform SQL-like queries on indexed cache data. This functionality becomes particularly beneficial when the values of the keys storing the desired information are not known. It abstracts much of the lower-level cache API calls, contributing to clearer and more maintainable application code. This feature is especially advantageous for individuals who find SQL-like commands more intuitive and comfortable to work with. NCache provides functionality for searching and removing cache data through queries similar to SQL's SELECT and DELETE statements. However, operations like INSERT and UPDATE are not available. For executing SELECT queries within the cache, NCache utilizes ExecuteReader; the ExecuteScalar function is used to carry out a query and retrieve the first row's first column from the resulting data set, disregarding any extra columns or rows. For NCache SQL queries to function, indexes must be established on all objects undergoing search. This can be achieved through two methods: configuring the cache or utilizing code with "Custom Attributes" to annotate object fields. When objects are added to the cache, this approach automatically creates indexes on the specified fields. Code Snippet Java String cacheName = "demoCache"; // Connect to the cache and return a cache handle Cache cache = CacheManager.getCache(cacheName); // Adds all the products to the cache. This automatically creates indexes on various // attributes of Product object by using "Custom Attributes". addSampleData(cache); // $VALUE$ keyword means the entire object instead of individual attributes that are also possible String sql = "SELECT $VALUE$ FROM com.alachisoft.ncache.samples.Product WHERE category IN (?, ?) AND price < ?"; QueryCommand sqlCommand = new QueryCommand(sql); List<String> catParamList = new ArrayList<>(Arrays.asList(("Electronics"), ("Stationery"))); sqlCommand.getParameters().put("category", catParamList); sqlCommand.getParameters().put("price", 2000); // ExecuteReader returns ICacheReader with the query resultset CacheReader resultSet = cache.getSearchService().executeReader(sqlCommand); List<Product> fetchedProducts = new ArrayList<>(); if (resultSet.getFieldCount() > 0) { while (resultSet.read()) { // getValue() with $VALUE$ keyword returns the entire object instead of just one column fetchedProducts.add(resultSet.getValue("$VALUE$", Product.class)); } } printProducts(fetchedProducts); Utilize SQL in NCache to perform queries on cached data by focusing on object attributes and Tags, rather than solely relying on keys. In this example, we utilize "Custom Attributes" to generate an index on the Product object. Learn more about SQL Query with NCache in Java Query Data in Cache Using SQL Read-Thru and Write-Thru Utilize the Data Source Providers feature of NCache to position it as the primary interface for data access within your microservices architecture. When a microservice needs data, it should first query the cache. If the data is present, the cache supplies it directly. Otherwise, the cache employs a read-thru handler to fetch the data from the datastore on behalf of the client, caches it, and then provides it to the microservice. In a similar fashion, for write operations (such as Add, Update, Delete), a microservice can perform these actions on the cache. The cache then automatically carries out the corresponding write operation on the datastore using a write-thru handler. Furthermore, you have the option to compel the cache to fetch data directly from the datastore, regardless of the presence of a possibly outdated version in the cache. This feature is essential when microservices require the most current information and complements the previously mentioned cache consistency strategies. The integration of the Data Source Provider feature not only simplifies your application code but also, when combined with NCache's database synchronization capabilities, ensures that the cache is consistently updated with fresh data for processing. ReadThruProvider For implementing Read-Through caching, it's necessary to create an implementation of the ReadThruProvider interface in Java Here's a code snippet to get started with implementing Read-Thru in your microservices: Java ReadThruOptions readThruOptions = new ReadThruOptions(ReadMode.ReadThru, _readThruProviderName); product = _cache.get(_productId, readThruOptions, Product.class); Read more about Read-Thru implementation here: Read-Through Provider Configuration and Implementation WriteThruProvider: For implementing Write-Through caching, it's necessary to create an implementation of the WriteThruProvider interface in Java The code snippet to get started with implementing Write-Thru in your microservices: Java _product = new Product(); WriteThruOptions writeThruOptions = new WriteThruOptions(WriteMode.WriteThru, _writeThruProviderName) CacheItem cacheItem= new CacheItem(_customer) _cache.insert(_product.getProductID(), cacheItem, writeThruOptions); Read more about Write-Thru implementation here: Write-Through Provider Configuration and Implementation Summary Microservices are designed to be autonomous, enabling independent development, testing, and deployment from other microservices. While microservices provide benefits in scalability and rapid development cycles, some components of the application stack can present challenges. One such challenge is the use of relational databases, which may not support the necessary scale-out to handle growing loads. This is where a distributed caching solution like NCache becomes valuable. In this article, we have seen the variety of ready-to-use features like pub/sub messaging, data caching, SQL Query, Read-Thru and Write-Thru, and Hibernate second-level Java Cache techniques offered by NCache that simplify and streamline the integration of data caching into your microservices application, making it an effortless and natural extension.

AWS re:Invent is an annual conference hosted by Amazon Web Services. AWS re:Invent 2023 stood out as a beacon of innovation, education, and vision in cloud computing. Held in Las Vegas, Nevada, spread over five days, the conference was one of the largest gatherings in the cloud sector, attracting an estimated 65,000+ attendees from around the globe. Having had the privilege to attend this year (2023), I am excited to share the key takeaways from the conference and interactions with some of the brightest minds in cloud computing. I aim to inspire and shed light on the expansive possibilities cloud technology offers. AWS Aurora Limitless Database In today’s world, enterprise applications typically rely on backend databases to host all the data necessary for the application. As you add new capabilities to your application or there is a growth in the customer base on your application, the volume of data hosted by the database surges rapidly, and the number of transactions that require database interaction increases significantly. There are many proven ways to manage this increased load to your database that can enhance the performance of the backing database. For example, we can scale up our database by allocating more vCPU and memory. Optimizing the SQL queries or using advanced features like “Input-Output optimized reads” from Amazon Aurora databases can significantly enhance the performance. We can also add additional read-only (read replicas) nodes/workers to support additional interaction from the database, which only requires read operation. However, before the AWS Aurora Limitless database launched, no out-of-the-box features were available that allowed data to be distributed across multiple database instances - a process known as database sharding. Sharding allows each instance to handle parallel write requests, significantly enhancing write operation performance. However, sharding requires the application team to add logic within the application to determine which database instance should serve that request. In addition, sharding also introduces enormous complexity, as the application must manage the ACID transactions and ensure consistency guarantees. Amazon Aurora Limitless Database addresses these challenges by handling the scalability of sharded databases with the simplicity of managing a single database. It also maintains transactional consistency across the system, which allows for handling millions of transactions per second and managing petabytes of data within a single Aurora cluster. As a consumer of the Amazon Aurora Limitless database, you only need to interact with a single database endpoint. The underlying architecture of Amazon Aurora Limitless ensures that write requests are directed to the appropriate database instance. Therefore, if your use case involves processing millions of write requests per second, Amazon Aurora Limitless Database is well-equipped to meet this demand effortlessly. Amazon S3 Express Zone Amazon S3 Express Zone is a single Availability Zone storage class that consistently delivers single-digit millisecond data access for frequently accessed data. When compared to S3 Standard, it delivers data access speed up to 10x faster and request costs up to 50% lower. Amazon S3 Express One Zone is ideal for use cases where you need high performance, low latency, and cost-effective storage solutions while not requiring the multi-availability zone (AZ) data resiliency offered by other S3 storage classes. So, suppose you want to process large amounts of data quickly, such as scientific simulations, big data analytics, or training machine learning models. In that case, S3 Express One Zone supports these intensive workloads by enabling faster data feeding to computation engines. ElastiCache Serverless Before learning more about ElastiCache Serverless, it's essential to understand the role of caching in modern applications. A cache is an in-memory data storage that enables applications to access data quickly, with high speed and low latency, significantly enhancing web applications' performance. Amazon ElastiCache, provided by Amazon Web Services, is a fully managed in-memory data store and caching service compatible with open-source in-memory data stores, such as Redis and Memcached. In the traditional ElastiCache setup, we need to specify the capacity of the ElastiCache cluster upfront while creating the cluster. This capacity remains fixed, leading to potential throttling if demand exceeds this capacity or wasted resources if the demand is consistently below capacity. While it's possible to manually scale resources or implement custom scaling solutions, managing this for applications with continuous, variable traffic can be complex and cumbersome. In contrast, ElastiCache Serverless is a fully managed service from AWS, which eliminates the need for manual capacity management. This serverless model automatically allows horizontal and vertical scaling to match traffic demand without affecting application performance. It continuously monitors the CPU, memory, and network utilization of the ElastiCache cluster to dynamically scale cluster capacity in or out to align with the current demand, ensuring optimal efficiency and performance. ElastiCache Serverless maintains a warm pool of engine nodes, allowing it to add resources on the fly and meet changing demand seamlessly and reasonably quickly. And, since it's a managed service from AWS, we don't have to worry about software updates, as they are handled automatically by AWS. In addition, you pay only for the capacity you use. This can enable cost savings compared to provisioning for peak capacity, especially for workloads with variable traffic patterns. Finally, launching a serverless Elasticache cluster is extremely quick; it can be created within a minute via the AWS console. Amazon Q Amazon Q, launched during AWS: reInvent 2023, is a Generative AI-driven service built to assist IT specialists and developers in navigating the complexities of the entire application development cycle, which includes initial research, development, deployment, and maintenance phases. It integrates seamlessly with your enterprise information repositories and codebases, enabling the generation of content and actions based on enterprise system data. Amazon Q also facilitates the selection of optimal instance types for specific workloads, leading to cost-effective deployment strategies. Additionally, Amazon Q simplifies error resolution across AWS services by providing quick insights without requiring manual log reviews or in-depth research. Furthermore, Amazon Q addresses network connectivity challenges using tools like the Amazon VPC Reachability Analyzer to pinpoint and correct potential network misconfiguration. Its integration with development environments through Amazon CodeWhisperer further enhances its utility, allowing developers to ask questions and receive code explanations and optimizations. This feature is especially beneficial for debugging, testing, and developing new features. While Amazon Q can address a broad spectrum of challenges throughout the application development lifecycle, its capabilities extend far beyond the scope of this article. Machine Learning Capabilities Offered by CloudWatch Amazon CloudWatch is an AWS monitoring service that collects logs, metrics, and events, providing insights into AWS resources and applications. It has been enhanced with machine learning capabilities, which include pattern analysis, comparison analysis, and anomaly detection for efficient log data analysis. The recent introduction of a generative AI feature that generates Logs Insight queries from natural language prompts further simplifies log analysis for cloud users. For a detailed exploration of these features, please refer to this article: Effective Log Data Analysis with Amazon CloudWatch. Additional Highlights from AWS re:Invent 2023 There are several other notable highlights from AWS re:Invent 2023, including Zero ETL integrations with OpenSearch Service, which simplifies data analysis by enabling direct, seamless data transfers without creating complex ETL processes. AWS Glue, a serverless ETL service, added anomaly detection features for improved data quality, and Application Load Balancer now supports automatic target weights based on health indicators like HTTP 500 errors. To explore a full rundown of announcements and in-depth analyses, please see the AWS Blog. Conclusion AWS re:Invent 2023 offered a unique opportunity to dive deep into the cloud technologies shaping our world. It highlighted the path forward in cloud technology, showcasing many innovations and insights. The conference underscores the endless possibilities that AWS continues to unlock for developers, IT professionals, and businesses worldwide.

Managing your secrets well is imperative in software development. It's not just about avoiding hardcoding secrets into your code, your CI/CD configurations, and more. It's about implementing tools and practices that make good secrets management almost second nature. A Quick Overview of Secrets Management What is a secret? It's any bit of code, text, or binary data that provides access to a resource or data that should have restricted access. Almost every software development process involves secrets: credentials for your developers to access your version control system (VCS) like GitHub, credentials for a microservice to access a database, and credentials for your CI/CD system to push new artifacts to production. There are three main elements to secrets management: How are you making them available to the people/resources that need them? How are you managing the lifecycle/rotation of your secrets? How are you scanning to ensure that the secrets are not being accidentally exposed? We'll look at elements one and two in terms of the secrets managers in this article. For element three, well, I'm biased toward GitGuardian because I work there (disclaimer achieved). Accidentally exposed secrets don't necessarily get a hacker into the full treasure trove, but even if they help a hacker get a foot in the door, it's more risk than you want. That's why secrets scanning should be a part of a healthy secrets management strategy. What To Look for in a Secrets Management Tool In the Secrets Management Maturity Model, hardcoding secrets into code in plaintext and then maybe running a manual scan for them is at the very bottom. Manually managing unencrypted secrets, whether hardcoded or in a .env file, is considered immature. To get to an intermediate level, you need to store them outside your code, encrypted, and preferably well-scoped and automatically rotated. It's important to differentiate between a key management system and a secret management system. Key management systems are meant to generate and manage cryptographic keys. Secrets managers will take keys, passwords, connection strings, cryptographic salts, and more, encrypt and store them, and then provide access to them for personnel and infrastructure in a secure manner. For example, AWS Key Management Service (KMS) and AWS Secrets Manager (discussed below) are related but are distinct brand names for Amazon. Besides providing a secure way to store and provide access to secrets, a solid solution will offer: Encryption in transit and at rest: The secrets are never stored or transmitted unencrypted. Automated secrets rotation: The tool can request changes to secrets and update them in its files in an automated manner on a set schedule. Single source of truth: The latest version of any secret your developers/resources need will be found there, and it is updated in real-time as keys are rotated. Role/identity scoped access: Different systems or users are granted access to only the secrets they need under the principle of least privilege. That means a microservice that accesses a MongoDB instance only gets credentials to access that specific instance and can't pull the admin credentials for your container registry. Integrations and SDKs: The service has APIs with officially blessed software to connect common resources like CI/CD systems or implement access in your team's programming language/framework of choice. Logging and auditing: You need to check your systems periodically for anomalous results as a standard practice; if you get hacked, the audit trail can help you track how and when each secret was accessed. Budget and scope appropriate: If you're bootstrapping with 5 developers, your needs will differ from those of a 2,000-developer company with federal contracts. Being able to pay for what you need at the level you need it is an important business consideration. The Secrets Manager List Cyberark Conjur Secrets Manager Enterprise Conjur was founded in 2011 and was acquired by Cyberark in 2017. It's grown to be one of the premiere secrets management solutions thanks to its robust feature set and large number of SDKs and integrations. With Role Based Access Controls (RBAC) and multiple authentication mechanisms, it makes it easy to get up and running using existing integrations for top developer tools like Ansible, AWS CloudFormation, Jenkins, GitHub Actions, Azure DevOps, and more. You can scope secrets access to the developers and systems that need the secrets. For example, a Developer role that accesses Conjur for a database secret might get a connection string for a test database when they're testing their app locally, while the application running in production gets the production database credentials. The Cyberark site boasts an extensive documentation set and robust REST API documentation to help you get up to speed, while their SDKs and integrations smooth out a lot of the speed bumps. In addition, GitGuardian and CyberArk have partnered to create a bridge to integrate CyberArk Conjur and GitGuardian's Has My Secrets Leaked. This is now available as an open-source project on GitHub, providing a unique solution for security teams to detect leaks and manage secrets seamlessly. Google Cloud Secret Manager When it comes to choosing Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure (Azure), it's usually going to come down to where you're already investing your time and money. In a multi-cloud architecture, you might have resources spread across the three, but if you're automatically rotating secrets and trying to create consistency for your services, you'll likely settle on one secrets manager as a single source of truth for third-party secrets rather than spreading secrets across multiple services. While Google is behind Amazon and Microsoft in market share, it sports the features you expect from a service competing for that market, including: Encryption at rest and in transit for your secrets CLI and SDK access to secrets Logging and audit trails Permissioning via IAM CI/CD integrations with GitHub Actions, Hashicorp Terraform, and more. Client libraries for eight popular programming languages. Again, whether to choose it is more about where you're investing your time and money rather than a killer function in most cases. AWS Secrets Manager Everyone with an AWS certification, whether developer or architect, has heard of or used AWS Secrets Manager. It's easy to get it mixed up with AWS Key Management System (KMS), but the Secrets Manager is simpler. KMS creates, stores, and manages cryptographic keys. Secrets Manager lets you put stuff in a vault and retrieve it when needed. A nice feature of AWS Secrets Manager is that it can connect with a CI/CD tool like GitHub actions through OpenID Connect (OIDC), and you can create different IAM roles with tightly scoped permissions, assigning them not only to individual repositories but specific branches. AWS Secrets Manager can store and retrieve non-AWS secrets as well as use the roles to provide access to AWS services to a CI/CD tool like GitHub Actions. Using AWS Lambda, key rotation can be automated, which is probably the most efficient way, as the key is updated in the secrets manager milliseconds after it's changed, producing the minimum amount of disruption. As with any AWS solution, it's a good idea to create multi-region or multi-availability-zone replicas of your secrets, so if your secrets are destroyed by a fire or taken offline by an absent-minded backhoe operator, you can fail over to a secondary source automatically. At $0.40 per secret per month, it's not a huge cost for added resiliency. Azure Key Vault Azure is the #2 player in the cloud space after AWS. Their promotional literature touts their compatibility with FIPS 140-2 standards and Hardware Security Modules (HSMs), showing they have a focus on customers who are either government agencies or have business with government agencies. This is not to say that their competitors are not suitable for government or government-adjacent solutions, but that Microsoft pushes that out of the gate as a key feature. Identity-managed access, auditability, differentiated vaults, and encryption at rest and in transit are all features they share with competitors. As with most Microsoft products, it tries to be very Microsoft and will more than likely appeal more to .Net developers who use Microsoft tools and services already. While it does offer a REST API, the selection of officially blessed client libraries (Java, .Net, Spring, Python, and JavaScript) is thinner than you'll find with AWS or GCP. As noted in the AWS and GCP entries, a big factor in your decision will be which cloud provider is getting your dominant investment of time and money. And if you're using Azure because you're a Microsoft shop with a strong investment in .Net, then the choice will be obvious. Doppler While CyberArk's Conjur (discussed above) started as a solo product that was acquired and integrated into a larger suite, Doppler currently remains a standalone key vault solution. That might be attractive for some because it's cloud-provider agnostic, coding language agnostic, and has to compete on its merits instead of being the default secrets manager for a larger package of services. It offers logging, auditing, encryption at rest and in transit, and a list of integrations as long as your arm. Besides selling its abilities, it sells its SOC compliance and remediation functionalities on the front page. When you dig deeper, there's a list of integrations as long as your arm testifies to its usefulness for integrating with a wide variety of services, and its list of SDKs is more robust than Azure's. It seems to rely strongly on injecting environment variables, which can make a lot of your coding easier at the cost of the environment variables potentially ending up in run logs or crash dumps. Understanding how the systems with which you're using it treat environment variables, scope them, and the best ways to implement it with them will be part of the learning curve in adopting it. Infisical Like Doppler, Infisical uses environment variable injection. Similar to the Dotenv package for Node, when used in Node, it injects them at run time into the process object of the running app so they're not readable by any other processes or users. They can still be revealed by a crash dump or logging, so that is a caveat to consider in your code and build scripts. Infisical offers other features besides a secrets vault, such as configuration sharing for developer teams and secrets scanning for your codebase, git history, and as a pre-commit hook. You might ask why someone writing for GitGuardian would mention a product with a competing feature. Aside from the scanning, their secrets and configuration vault/sharing model offers virtual secrets, over 20 cloud integrations, nine CI/CD integrations, over a dozen framework integrations, and SDKs for four programming languages. Their software is mostly open-source, and there is a free tier, but features like audit logs, RBAC, and secrets rotation are only available to paid subscribers. Akeyless AKeyless goes all out features, providing a wide variety of authentication and authorization methods for how the keys and secrets it manages can be accessed. It supports standards like RBAC and OIDC as well as 3rd party services like AWS IAM and Microsoft Active Directory. It keeps up with the competition in providing encryption at rest and in transit, real-time access to secrets, short-lived secrets and keys, automated rotation, and auditing. It also provides features like just-in-time zero trust access, a password manager for browser-based access control as well as password sharing with short-lived, auto-expiring passwords for third parties that can be tracked and audited. In addition to 14 different authentication options, it offers seven different SDKs and dozens of integrations for platforms ranging from Azure to MongoDB to Remote Desktop Protocol. They offer a reasonable free tier that includes 3-days of log retention (as opposed to other platforms where it's a paid feature only). 1Password You might be asking, "Isn't that just a password manager for my browser?" If you think that's all they offer, think again. They offer consumer, developer, and enterprise solutions, and what we're going to look at is their developer-focused offering. Aside from zero-trust models, access control models, integrations, and even secret scanning, one of their claims that stands out on the developer page is "Go ahead – commit your .env files with confidence." This stands out because .env files committed to source control are a serious source of secret sprawl. So, how are they making that safe? You're not putting secrets into your .env files. Instead, you're putting references to your secrets that allow them to be loaded from 1Password using their services and access controls. This is somewhat ingenious as it combines a format a lot of developers know well with 1Password's access controls. It's not plug-and-play and requires a bit of a learning curve, but familiarity doesn't always breed contempt. Sometimes it breeds confidence. While it has a limited number of integrations, it covers some of the biggest Kubernetes and CI/CD options. On top of that, it has dozens and dozens of "shell plugins" that help you secure local CLI access without having to store plaintext credentials in ~/.aws or another "hidden" directory. And yes, we mentioned they offer secrets scanning as part of their offering. Again, you might ask why someone writing for GitGuardian would mention a product with a competing feature. HashiCorp Vault HashiCorp Vault offers secrets management, key management, and more. It's a big solution with a lot of features and a lot of options. Besides encryption, role/identity-based secrets access, dynamic secrets, and secrets rotation, it offers data encryption and tokenization to protect data outside the vault. It can act as an OIDC provider for back-end connections as well as sporting a whopping seventy-five integrations in its catalog for the biggest cloud and identity providers. It's also one of the few to offer its own training and certification path if you want to add being Hashi Corp Vault certified to your resume. It has a free tier for up to 25 secrets and limited features. Once you get past that, it can get pricey, with monthly fees of $1,100 or more to rent a cloud server at an hourly rate. In Summary Whether it's one of the solutions we recommended or another solution that meets our recommendations of what to look for above, we strongly recommend integrating a secret management tool into your development processes. If you still need more convincing, we'll leave you with this video featuring GitGuardian's own Mackenzie Jackson.

In this tutorial, we will learn how to build and deploy a conversational chatbot using Google Cloud Run and Dialogflow. This chatbot will provide responses to user queries on a specific topic, such as weather information, customer support, or any other domain you choose. We will cover the steps from creating the Dialogflow agent to deploying the webhook service on Google Cloud Run. Prerequisites A Google Cloud Platform (GCP) account Basic knowledge of Python programming Familiarity with Google Cloud Console Step 1: Set Up Dialogflow Agent Create a Dialogflow Agent: Log into the Dialogflow Console (Google Dialogflow). Click on "Create Agent" and fill in the agent details. Select the Google Cloud Project you want to associate with this agent. Define Intents: Intents classify the user's intentions. For each intent, specify examples of user phrases and the responses you want Dialogflow to provide. For example, for a weather chatbot, you might create an intent named "WeatherInquiry" with user phrases like "What's the weather like in Dallas?" and set up appropriate responses. Step 2: Develop the Webhook Service The webhook service processes requests from Dialogflow and returns dynamic responses. We'll use Flask, a lightweight WSGI web application framework in Python, to create this service. Set Up Your Development Environment: Ensure you have Python and pip installed. Create a new directory for your project and set up a virtual environment: Shell python -m venv env source env/bin/activate # `env\Scripts\activate` for windows Install Dependencies: Install Flask and the Dialogflow library: Shell pip install Flask google-cloud-dialogflow Create the Flask App: In your project directory, create a file named app.py. This file will contain the Flask application: Python from flask import Flask, request, jsonify app = Flask(__name__) @app.route('/webhook', methods=['POST']) def webhook(): req = request.get_json(silent=True, force=True) # Process the request here. try: query_result = req.get('queryResult') intent_name = query_result.get('intent').get('displayName') response_text = f"Received intent: {intent_name}" return jsonify({'fulfillmentText': response_text}) except AttributeError: return jsonify({'fulfillmentText': "Error processing the request"}) if __name__ == '__main__': app.run(debug=True) Step 3: Deploy To Google Cloud Run Google Cloud Run is a managed platform that enables you to run containers statelessly over a fully managed environment or in your own Google Kubernetes Engine cluster. Containerize the Flask App: Create a Dockerfile in your project directory: Dockerfile FROM python:3.8-slim WORKDIR /app COPY requirements.txt requirements.txt RUN pip install -r requirements.txt COPY . . CMD ["flask", "run", "--host=0.0.0.0", "--port=8080"] Don't forget to create a requirements.txt file listing your Python dependencies: Flask==1.1.2 google-cloud-dialogflow==2.4.0 Build and Push the Container: Use Cloud Build to build your container image and push it to the container registry. Shell gcloud builds submit --tag gcr.io/YOUR_CHATBOT_PRJ_ID/chatbot-webhook . Deploy to Cloud Run: Deploy your container image to Cloud Run. Shell gcloud run deploy --image gcr.io/YOUR_PROJECT_ID/chatbot-webhook --platform managed Follow the prompts to enable the required APIs, choose a region, and allow unauthenticated invocations. Step 4: Integrate With Dialogflow In the Dialogflow Console, navigate to the Fulfillment section. Enable Webhook, paste the URL of your Cloud Run service (you get this URL after deploying to Cloud Run), and click "Save." Testing and Iteration Test your chatbot in the Dialogflow Console's simulator. You can refine your intents, entities, and webhook logic based on the responses you receive. Conclusion You have successfully built and deployed a conversational chatbot using Google Cloud Run and Dialogflow. This setup allows you to create scalable, serverless chatbots that can handle dynamic responses to user queries. This foundation allows for further customization and expansion, enabling the development of more complex and responsive chatbots to meet a variety of needs. Continue to refine your chatbot by adjusting intents, entities, and the webhook logic to improve interaction quality and user experience.

By Ashok Gorantla CORE