Grok AI on AWS EC2

For running AI models such as Llama 3 70B with quantization, the g6.xlarge EC2 instance is recommended. This instance features:

• 1 NVIDIA L4 GPU with 24 GB VRAM, suitable for quantized large language models (LLMs).
• 4 vCPUs and 16 GB RAM, sufficient for development tasks.
• Cost: Approximately $0.8048 per hour in US East (N. Virginia), as per Vantage.

The L4 GPU, based on NVIDIA's Ada Lovelace architecture, offers comparable performance to the A10G GPU (in g5.xlarge) for memory-bound inference tasks, but at a lower cost (~20% less than g5.xlarge's $1.006/hour). The 24 GB VRAM supports models like Llama 3 70B when quantized to 4-bit, which reduces memory needs to ~35 GB or less with optimization, fitting within the GPU's capacity.

To deploy the AI agent, follow these steps:

1. Launch an EC2 Instance: Select the g6.xlarge instance with the Deep Learning AMI (Ubuntu) from the AWS Marketplace. Ensure the "Shutdown behavior" is set to "Stop" in the EC2 console to enable cost-saving shutdowns.
2. Connect to the Instance: Use SSH to access the instance (e.g., ssh -i your-key.pem ubuntu@instance-ip).
3. Run the Setup Script: Copy and execute the provided script to install and configure all components.

Setup Script

Script Explanation

Nginx serves Open WebUI, proxying requests from port 80 to port 3000. The configuration includes security headers to mitigate common web vulnerabilities. If you have a domain, Certbot automates SSL setup, enabling HTTPS access.

To ensure services run reliably and restart after reboots:

• Ollama: Installed as a systemd service, automatically starting and restarting on failure.
• Open WebUI: Docker container with --restart always policy, ensuring uptime.
• Code-Server: Configured as a systemd service for consistent operation.

Verify service status with systemctl status ollama, docker ps, and systemctl status code-server.

Code-Server provides a browser-based Visual Studio Code environment, accessible at http://your-instance-ip:8080. Set a secure password in the script's CODE_SERVER_PASSWORD variable.

To optimize costs, the script includes a cron job that stops the instance at 2 AM daily, assuming the EC2 "Shutdown behavior" is set to "Stop." This reduces runtime to ~8 hours/day, saving ~60% compared to 24/7 operation.

To restart, use the AWS console or AWS CLI (aws ec2 start-instances). For more advanced scheduling, consider AWS Instance Scheduler, though the cron job is simpler for basic needs.

The following table outlines estimated monthly costs for the g6.xlarge setup in US East (N. Virginia):

• Model Support: Verify model availability in Ollama's library. For unavailable models (e.g., Qwen2.5-Coder), import GGUF files manually.
• Security: HTTP access without SSL is less secure. A domain enables HTTPS, improving safety.
• Scalability: For heavier workloads, consider g6.12xlarge (4 GPUs, 96 GB VRAM, ~$4.83/hour).
• Monitoring: Use AWS CloudWatch to track usage and set cost alerts.

This setup delivers a cost-effective, self-hosted AI agent on AWS EC2, avoiding subscription fees while providing cloud-like accessibility. The g6.xlarge instance, combined with Ollama, Open WebUI, and Code-Server, offers a powerful platform for running AI models and developing remotely.

Automated cost management ensures affordability, with monthly costs as low as $193 for moderate use. By following this guide, you can achieve a reliable, secure, and efficient AI development environment tailored to your needs.

• Amazon EC2 G6 Instances Specifications
• Amazon EC2 G5 Instances Specifications
• g6.xlarge Pricing and Specifications
• g5.xlarge Pricing and Specifications
• Ollama Linux Installation Guide
• Open WebUI Quick Start Guide
• Code-Server Installation Instructions
• NVIDIA L4 Tensor Core GPU Specifications
• NVIDIA A10 Tensor Core GPU Specifications