Jobs › Blog › Cloud subsystems
29 Aug 2026 · Graphai Jobs
AWS and GCP subsystems for apply and crawl
The apply fleet is a Playwright worker plus a small amount of managed cloud around it. Caps stay €100 / month on AWS and €200 spend / 200 VMs on GCP. This note lists every Amazon and Google subsystem that is live, planned, or deliberately off — with the official documentation for each.
Related: Apply fleet data architecture.
How to read the labels
- Live — provisioned or called from the apply/crawl path today.
- Planned — fits the caps; wired or sketched, not 24/7.
- Off — too expensive, wrong tool, or blocked by a hard constraint (no NAT/LB, no 24/7 AgentCore, Proton Bridge instead of SES, no LinkedIn Easy Apply).
Amazon Web Services
Account 507377784850, default apply region eu-central-1 (Frankfurt). One Spot worker: apply-aws-1, m7i-flex.large, MaxPrice $0.05. No NAT Gateway, no load balancer, no extra Elastic IPs.
Amazon EC2 Spot Live
Runs headless Chromium for career-site crawls and form apply. Spot is the only EC2 mode under the €100 envelope. The instance profile is apply-aws-ssm. Disk is 32 GB gp3, delete-on-termination. A 30-minute heartbeat watchdog shuts the VM down if apply stalls.
AWS Identity and Access Management Live
Instance role plus a tight inline policy: CloudWatch logs/metrics, S3 apply/*, Parameter Store /apply/aws*. API keys never live on the VM. Operator login is IAM Identity Center / aws login, not long-lived keys in the engine tree.
AWS Systems Manager Live
Session Manager and Run Command drive apply-aws-1 when this Mac’s WAN is a Tor exit and SSH/22 times out. Inventory, start run_worker.sh, and pull heartbeats go through SSM instead of opening the security group to the world.
Amazon CloudWatch Logs Live
Log group /apply/aws, 14-day retention, stream per host. The apply engine emits structured steps (ledger, protocol, submit / block / thank-you). Verbose debug lines stay off unless APPLY_CLOUDWATCH_VERBOSE=1.
Amazon CloudWatch Metrics Live
Namespace Apply/Jobbot with four custom metrics only: Attempts, Success, Fail, Blocked. No per-company dimensions — those would bill a custom metric per ATS.
AWS CloudTrail Live
Trail apply-mgmt: management events only, multi-region, log-file validation. This is the API audit log (who launched, stopped, or changed IAM). It is not a spend feed. First copy of management events is $0; data events stay off.
Amazon S3 Live
Private bucket apply-jobbot-507377784850-euc1: CloudTrail prefixes plus copies of ledgers and the 3-hour applications CSV. AES-256, public access blocked, 90-day lifecycle. No CVs or .env files.
Parameter Store (Systems Manager) Live
Standard-tier strings under /apply/aws/: queue shard IDX_ENV=202, APPLY_TOR_FOR_CV=0, log group, S3 bucket, €100 cap. Not Secrets Manager. Hunter/PDL keys stay off the VMs.
AWS Budgets Live
Budgets jobbot-90 ($90) and the account monthly $50 budget. AWS Budget SNS cannot talk to Proton Mail Bridge, so the Mac 3-hour poller reads the budget and emails martibayoalemany@gmail.com when spend crosses 80% of $90.
AWS Cost Explorer Live
Month-to-date UnblendedCost plus forecast. Same Proton alert path as Budgets. This is the spend API; CloudTrail is not.
Amazon Bedrock Planned
Pay-per-call vision fallback when Vertex generateContent returns 403. Default model: Claude 3 Haiku in eu-central-1 (Nova Lite is listed too). Used only on stuck apply screenshots, not on every crawl. Company briefs on the jobs studio already have a Bedrock path next to Vertex.
Amazon Bedrock AgentCore Off
Runtime microVMs / Browser could unstick a failing ATS page. Dedicated Runtime Instances (EC2 + 12% fee) and 24/7 AgentCore stay off under the €100 cap. Burst-only on failing screens remains optional after a fresh aws login, not a standing service.
AWS Lambda (Playwright container) Planned
Container image with Chromium, 3008 MB, 10-minute timeout, reserved concurrency 1, COMPLETE_MAX=2. This is the AWS twin of Cloud Run Jobs — one or two applies per invoke, not a 15-minute zip of a tiny runtime. Proton is not in Lambda. Deploy: aws_lambda/deploy.sh.
AWS Secrets Manager Planned
Portal host/user/password vault at apply/portal-credentials (gzip so it fits the 64 KiB cap). Hunter/PDL keys still stay off the VMs. Automation default remains NEVER_LOGIN=1; Lambda/EC2 can read the secret when a portal login is explicitly allowed. ~$0.40/secret/month.
Amazon EventBridge Planned
Rule apply-playwright-every-30m invokes the Playwright Lambda. First million events/month are free. EventBridge cannot talk to Proton Bridge, so cost-alert mail stays on the Mac poller. Spot-interruption notices are a possible extra target later.
AWS subsystems we are not using
| Subsystem | Why not | Docs |
|---|---|---|
| NAT Gateway / Elastic Load Balancing | Idle tens of euros; the worker has a public IP and a /32 SSH group. | NAT Gateway · ELB |
| Amazon SES | Reports stay on Proton Bridge 127.0.0.1:1025. |
SES |
| AWS Transform | Agentic migration of VMware, mainframe, and .NET to AWS. It does not crawl career sites or fill ATS forms. Left off so it cannot spend against the €100 apply envelope. | AWS Transform |
Google Cloud
Project poetic-sentinel-402405, region europe-west1. Spot fleet name prefix apply-spot-*, machine type e2-standard-2. Hard wipe at €200. Reuse VMs; do not spawn 200 extra instances.
Compute Engine Spot VMs Live
Primary crawl and apply workers. Same Playwright tree as AWS. Watchdog 2700s. Queue shards by IDX_ENV so GCP and AWS do not steal the same rows. This is the GCP analog of EC2 Spot.
Cloud Run jobs Live
Serverless apply slices (GCP analog of “Lambda, but long enough for a browser”). Smoke: 2 tasks, 2 vCPU / 4 GiB, 1 h timeout, COMPLETE_MAX=8. Proton Bridge is not in the container; mail stays on the Mac.
Vertex AI (Gemini) Live
Vision on failing apply screenshots and company briefs in the jobs studio. Model default: Gemini 2.5 Flash in europe-west1. If generateContent 403s, the engine falls back to Bedrock. Not called on every crawl.
Cloud Storage Live
Packs the apply-engine tree (no secrets) onto short-lived apply-h3-* VMs, then the VM deletes itself. Analog of the AWS S3 apply prefix, without putting CVs or API keys in the object.
BigQuery Live
Warehouse load of application rows and engine logs for the 3-hour report. Complements ClickHouse; does not replace the Mac CSV that is emailed via Proton.
Cloud Logging Live
GCP twin of CloudWatch Logs: worker stdout, fleet controller ticks, Cloud Run job lines. Redaction strips DSNs and API keys before ship.
IAM / service accounts Live
Compute and Vertex calls use a service account JSON in the operator environment (GCP_SA_JSON), never in the public jobs.graphai.eu repo. Scope is Compute plus cloud-platform for Vertex.
Cloud Functions (Gmail push ingest) Planned
Sketched path: Gmail push → function → queue_from_alerts.csv so first-day ATS posts land before the 3-hour Mac collect. LinkedIn Easy Apply remains refused; the function would only enqueue employer ATS URLs.
Vertex AI Agent Engine Planned
GCP analog of Bedrock AgentCore. Same rule: burst on a stuck form, never a 24/7 agent farm. Memory Bank / evals stay optional.
Cloud Trace Planned
Distributed traces for Cloud Run apply tasks and Vertex calls. Complements Cloud Logging; not a replacement for the honest SUCCESS = thank-you rule in the ledger.
GCP subsystems we are not using for apply
| Subsystem | Why not | Docs |
|---|---|---|
| Cloud Load Balancing / Cloud NAT | Same as AWS: idle cost, not needed for one public Spot VM. | Load Balancing · Cloud NAT |
| GKE | Overkill for Playwright workers; Spot VMs plus Cloud Run Jobs already split long vs short work. | GKE |
| Migrate to VMs / App Design Center | Migration products. Not crawl, not apply. | Migrate to VMs |
What actually crawls and applies
Neither cloud replaces Playwright. AWS and GCP only host the browser, store the ledger, score a stuck screenshot, and keep spend visible. Job discovery still comes from employer ATS APIs (Greenhouse, Lever, Ashby), career-page HTML, and Gmail JOB_ALERTS — never LinkedIn Easy Apply.