Model Garden of VertexAI
Model Garden of VertexAI:
Unlocking the Power of Google’s VertexAI: Exploring the World of Pre-Built Models for AI Tasks
Introduction:
Artificial Intelligence (AI) has transformed numerous industries, from healthcare and finance to e-commerce, logistic, eduction and entertainment. But the complexity of developing machine learning models often poses a challenge. As the demand for AI-powered solutions continues to rise, data scientists seek efficient ways to leverage pre-trained models or build custom models to address specific tasks. In this regard, Google’s VertexAI emerges as a robust platform that offers an extensive selection of pre-built models for a wide range of AI tasks. VertexAI platform has revolutionized the landscape by seamlessly leveraging LLM (Large Language Models) and Prompt Engineering techniques to perform complex machine learning tasks effortlessly. With VertexAI, data scientists can harness the power of state-of-the-art language models, such as LLM, to accelerate their ML development process. Additionally, the innovative concept of Prompt Engineering enables users to effectively communicate with the models, guiding them to deliver precise and accurate results. From computer vision and natural language processing to speech processing and structured tabular data analysis, Vertex AI’s repertoire includes over 100 models catering to diverse application domains. This article explores how Vertex AI, through its integration of LLM and Prompt Engineering, empowers users to effortlessly tackle intricate machine learning tasks across diverse domains, revolutionizing the AI development experience.
Foundation models:
Pre-trained multi-task models that can be further tuned or customized for specific tasks.
| Sno. | Name | Details | Task Name | Vision/ Language | Input DataType | Model Name |
|---|---|---|---|---|---|---|
| 1 | PaLM 2 for Text | Fine-tuned to follow natural language instructions and is suitable for a variety of language tasks, such as: classification, extraction, summarization and content generation. | Text Gen. | Language | Text | text-bison@001 |
| 2 | PaLM 2 for Chat | Fine-tuned to conduct natural conversation. Use this model to build and customize your own chatbot application. | Text Gen. | Language | Text | chat-bison@001 |
| 3 | Embeddings for text | Text embedding is an important NLP technique that converts textual data into numerical vectors that can be processed by machine learning algorithms, especially large models. These vector representations are designed to capture the semantic meaning and context of the words they represent. | Embedding | Language | Text | textembedding-gecko@001 |
| 4 | Codey for Code Completion | Generates code based on code prompts. Good for code suggestions and minimizing bugs in code. | Code Gen. | Language | Text | code-gecko@001 |
| 5 | Codey for Code Generation | Generates code based on natural language input. Good for writing functions, classes, unit tests, and more. | Code Gen. | Language | Text | code-bison@001 |
| 6 | Codey for Code Chat | Get code-related assistance through natural conversation. Good for questions about an API, syntax in a supported language, and more. | Code Gen. | Language | Text | codechat-bison@001 |
| 7 | BERT | Neural network-based technique for natural language processing. Use it to train your own question answering system and more. | Text Gen. | Language | Text | google/bert-base-001 |
| 8 | InstructPix2Pix | Given an input image and a text prompt that tells the model what to do, the instruct-pix2pix model follows the prompt to edit the image by generating a new one. | Image Gen. | Vision, Language | Text+Image | timbrooks/instruct-pix2pix |
| 9 | ControlNet | Control image generation with text prompt and control image. | Image Gen. | Vision, Language | Text | lllyasviel/ControlNet |
| 10 | BLIP2 | BLIP2 is for the image captioning and visual-question-answering tasks. | Text Gen. | Vision, Language | Image | Salesforce/blip2-opt-2.7b |
| 11 | Stable Diffusion 1.4 (Keras) | KerasCV implementation of stability.ai’s text-to-image model, Stable Diffusion 1.4. | Image Gen. | Vision, Language | Text | keras/stable-diffusion-v1-4 |
| 12 | Embeddings for Image | Generates vectors based on images, which can be used for downstream tasks like image classification, image search, and so on. | Embedding | Vision, | Image | imageembedding-001 |
| 13 | Label detector (PaLI zero-shot) | Label Detector Zero-shot classifies images based on labels, represented as a list of text prompt strings, which are provided by the user, and calculates the confidence score of each labelâs presence in the image. | Classification | Vision, | Image | imagezeroshot-001 |
| 14 | Stable Diffusion v1-5 | Latent text-to-image diffusion model capable of generating photo-realistic images given a text input. | Image Gen. | Vision, | Text | runwayml/stable-diffusion-v1-5 |
| 15 | Stable Diffusion Inpainting | Stable Diffusion Inpainting is a latent diffusion model capable of inpainting images given any text input and a mask image. | Image Gen. | Vision, | Text | runwayml/stable-diffusion-inpainting |
| 16 | BLIP image captioning | A Vision-Language Pre-training (VLP) framework for image captioning. | Text Gen. | Vision, | Image | Salesforce/blip-image-captioning-base |
| 17 | BLIP VQA | A Vision-Language Pre-training (VLP) framework for visual question answering (VQA). | Text Gen. | Vision, | Image | Salesforce/blip-vqa-base |
| 18 | CLIP | Neural network capable of classifying images without prior training on the classes. | Classification | Vision, | Image | openai/clip-vit-base-patch32 |
| 19 | OWL-ViT | Zero-shot, text-conditioned object detection model that can query an image with one or multiple text queries. | Text Gen. | Vision, | Text+Image | google/owlvit-base-patch32 |
| 20 | ViT GPT2 | Image captioning model | Text Gen. | Vision, | Image | nlpconnect/vit-gpt2-image-captioning |
| 21 | ViLT VQA | Vision-and-Language Transformer (ViLT) model fine-tuned on VQAv2. | Text Gen. | Vision, | Image | dandelin/vilt-b32-finetuned-vqa |
| 22 | LayoutLM for VQA | Fine-tuned for document understanding and information extraction tasks like form and receipt understanding. | Info. Extraction | Vision, | Scan Doc | impira/layoutlm-document-qa |
| 23 | T5-FLAN | T5 (Text-To-Text Transfer Transformer) model with the T5-FLAN checkpoint. | Text Gen. | Language | Text | google/t5-flan-001 |
| 24 | Sec-PaLM2 | The sec-palm model is a foundational model that has been pretrained on a variety of security-specific tasks. The model has broad security understanding across a number topics, such as threat intelligence, security operations, and malware analysis. It is ideal for analyzing, summarizing, and aggregating information across multiple security data sources, as well as generating rules and search queries from natural language input. | Info. Extraction | Language | Text | google/sec-palm-000 |
| 25 | Chirp | Chirp is a version of a Universal Speech Model that has over 2B parameters and can transcribe in over 100 languages in a single model. | Speech Gen. | Speech | chirp-rnnt1 |
Fine-tunable models :
Models that data scientists can further fine-tune through a custom notebook or pipeline.
| Sno. | Name | Details | Task Name | Vision/ Language | Input DataType | Model Name |
|---|---|---|---|---|---|---|
| 1 | Stable Diffusion Inpainting | Stable Diffusion Inpainting is a latent diffusion model capable of inpainting images given any text input and a mask image. | Image Gen. | Vision, Language | Text | runwayml/stable-diffusion-inpainting |
| 2 | ControlNet | Control image generation with text prompt and control image. | Image Gen. | Vision, | Text+Image | lllyasviel/ControlNet |
| 3 | tfhub/EfficientNetV2 | EfficientNet V2 are a family of image classification models, which achieve better parameter efficiency and faster training speed than prior arts. | Classification | Vision, | Image | tensorflow-hub/efficientnetv2 |
| 4 | tfvision/vit | The Vision Transformer (ViT) is a transformer-based architecture for image classification. | Classification | Vision, | Image | tfvision/vit-s16 |
| 5 | tfvision/SpineNet | SpineNet is an image object detection model generated using Neural Architecture Search. | Detection | Vision, | Image | tfvision/spinenet49 |
| 6 | tfvision/YOLO | YOLO algorithm is a one-stage object detection algorithm that can achieve real-time performance on a single GPU. | Detection | Vision, | Image | tfvision/scaled-yolo |
| 7 | DeepLabv3+ (with checkpoint) | Semantic segmentation is the task of assigning a label to each pixel in an image, where each label corresponds to a specific class of object or scene element. | Segmentation | Vision, | Image | deeplabv3plus-cityscapes-20230315 |
| 8 | ResNet (with checkpoint) | Image classification model as described in the paper “Deep Residual Learning for Image Recognition”. | Classification | Vision, | Image | resnet50 |
| 9 | ResNet-RS (with checkpoint) | Image classification model as described in the paper “Revisiting ResNets: Improved Training and Scaling Strategies”. | Classification | Vision, | Image | ResNet-RS-50 |
| 10 | Faster R-CNN (Detectron2) | Faster R-CNN is a deep convolutional network used for image object detection. | Detection | Vision, | Image | detectron2/faster-r-cnn |
| 11 | MobileNet (TIMM) | Small but powerful models optimized for mobile and embedded vision applications. | Classification | Vision, | Image | timm/mobilenetv2_100 |
| 12 | EfficientNet (TIMM) | A family of convolutional neural networks (CNNs) designed to be both accurate and efficient. | Classification | Vision, | Image | timm/efficientnetv2_rw_s |
| 13 | DeiT | A convolution-free transformer for image classification. | Classification | Vision, | Image | timm/deit_base_patch16_224 |
| 14 | BEiT | A self-supervised learning framework for image representation learning inspired by BERT. | Classification | Vision, | Image | timm/beit_base_patch16_224 |
| 15 | ViT (TIMM) | Transformer-like architecture for image classification. | Classification | Vision, | Image | timm/vit_base_patch16_224 |
| 16 | RetinaNet (Detectron2) | RetinaNet is a one-stage object detection model that utilizes a feature pyramid network (FPN) on top of a ResNet and adds a focal loss function to address class imbalance during training. | Detection | Vision, | Image | detectron2/retinanet |
| 17 | Mask R-CNN (Detectron2) | Mask R-CNN is an instance segmentation model which extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. | Detection | Vision, | Image | detectron2/mask-r-cnn |
| 18 | ResNet (TIMM) | A type of artificial neural network that is made up of residual blocks with skip connections. | Classification | Vision, | Image | timm/resnet50 |
| 19 | ResNeSt (TIMM) | An extension of the ResNet architecture that uses a new attention mechanism called split-attention. | Classification | Vision, | Image | timm/resnest50d |
| 20 | ConvNeXt (TIMM) | A pure convolutional model that is an extension of the ResNet architecture that uses a new attention mechanism called Swin Transformer. | Classification | Vision, | Image | timm/convnext_base |
| 21 | CspNet (TIMM) | A type of deep neural network that is an extension of the ResNet architecture that uses a new cross stage partial connection to reduce the number of parameters and computation cost without sacrificing accuracy. | Classification | Vision, | Image | timm/cspdarknet53 |
| 22 | Inception (TIMM) | Inception network is a deep neural network with an architectural design that consists of repeating components referred to as Inception modules. | Classification | Vision, | Image | timm/inception_v4 |
Task-specific solutions:
Most of these pre-built models are ready to use off the shelf, and many can be customized using your own data.
| Sno. | Name | Details | Task Name | Vision/ Language | Input DataType | Model Name |
|---|---|---|---|---|---|---|
| 1 | Entity analysis | Inspect text to identify and label persons, organizations, locations, events, products and more. | Classification | Language | Text | google/language_v1-analyze_entities |
| 2 | Content classification | Use Google’s state-of-the-art language technology to analyzes text content and returns content categories for the content. The latest version of Content Classification supports over 1,000 categories. | Classification | Language | Text | google/language_v1-classify_text_v1 |
| 3 | Sentiment analysis | Sentiment analysis attempts to determine the overall attitude (positive or negative) expressed within the text. Sentiment is represented by numerical score and magnitude values. | Classification | Language | Text | google/language_v1-analyze_sentiment |
| 4 | Entity sentiment analysis | Entity Sentiment Analysis inspects the given text for known entities (proper nouns and common nouns), returns information about those entities, and identifies the prevailing emotional opinion of the entity within the text, especially to determine a writer’s attitude toward the entity as positive, negative, or neutral. | Classification | Language | Text | google/language_v1-analyze_entity_sentiment |
| 5 | Syntax analysis | Syntactic analysis extracts linguistic information, breaking up the given text into a series of sentences and tokens (generally, word boundaries), providing further analysis on those tokens. | Extraction | Language | Text | google/language_v1-analyze_syntax |
| 6 | Text Moderation | Text moderation analyzes a document and returns a list of harmful and sensitive categories that apply to the text found in the document. | Classification | Language | Text | google/language_v1-moderate_text |
| 7 | Text Translation | Use Google’s proven pre-trained text model to get text translations for 100+ languages. | Translation | Language | Text | Text Translation |
| 8 | Occupancy analytics | Detect people and vehicles in a video or image, plus zone detection, dwell time, and more. | Detection | Vision, | Image, Video | google/occupancy-analytics-001 |
| 9 | Person/vehicle detector | Detects and counts people and vehicles in video. | Detection | Vision, | Video | People/vehicle detector |
| 10 | Object detector | Identify and locate objects in video | Detection | Vision, | Video | Object detector |
| 11 | PPE detector | Identify people and personal protective equipment (PPE). | Detection | Vision, | Image | PPE detector |
| 12 | Person blur | Mask or blur a person’s appearance in video | Detection | Vision, | Video | People blur |
| 13 | Product recognizer | Identify products at the GTIN or UPC level | Recognition | Vision, | Image | Product recognizer |
| 14 | Tag recognizer | Extract text in product and price tags | Recognition | Vision, | Scan Doc | Tag recognizer |
| 15 | Content moderation (Vision) | Content Moderator (Vision) detects objectionable or unwanted content across predefined content labels (e.g., adult, violence, spoof) or custom labels provided by the user. | Classification | Vision, | Scan Doc | Content Moderation |
| 16 | Face detector (Vision API) | Face detector is a prebuilt Vision API model that detects multiple faces in media (images, video) and provides bounding polygons for the face and other facial “landmarks” along with their corresponding confidence values. | Detection | Vision, | Image, Video | Face Detector |
| 17 | Watermark detector | Watermark detector is a prebuilt model that detects watermarks in the input image. | Detection | Vision, | Scan Doc | imagewatermarkdetector-001 |
| 18 | Text detector (Vision API) | Text detector detects and extracts text from images. It uses optical character recognition (OCR) for an image to recognize text and convert it to machine coded text. | Detection | Vision, Language | Scan Doc | Text Detector |
| 19 | AutoML E2E | Tabular Workflow for End-to-End AutoML is the complete AutoML pipeline for classification and regression tasks. | Classification | Tabular | AutoML E2E | |
| 20 | Document AI OCR processor | Document OCR can identify and extract text from documents in over 200 printed languages and 50 handwritten languages. | Extraction | Document | pretrained-ocr-v1.2-2022-11-10 | |
| 21 | Form Parser | Document AI Form Parser applies advanced machine learning technologies to extract key-value pairs, checkboxes, tables from documents in over 200+ languages. | Extraction | Document | pretrained-form-parser-v1.0-2020-09-23 | |
| 22 | TabNet | TabNet is a general model which performs well on a wide range of classification and regression tasks. | Classification | Tabular | TabNet |
Task-specific LLM Prompts :
Customize language model outputs to meet specific needs. Prompts help to refine or enrich the outputs of the large language model selected.
| Sno. | Name | Details | Task Name | Vision/ Language | Input DataType | Model Name |
|---|---|---|---|---|---|---|
| 1 | Object classification | Classify an object using a small number of examples (few-shot prompting). | Classification | Vision, | Structured | LLM Prompt |
| 2 | Kindergarten Science Teacher | Your name is Miles. You are an astronomer who is knowledgeable about the solar system. Respond in short sentences. Shape your response as if talking to a 10-years-old. | Text Gen. | Language | Freeform | LLM Prompt |
| 3 | Online Return Customer Service | A customer service chatbot that provides basic customer support and makes decisions on simple tasks | Text Gen. | Language | Freeform | LLM Prompt |
| 4 | Gluten Free Advisor | A chatbot that provides gluten free cooking recipes and diet plans. | Text Gen. | Language | Freeform | LLM Prompt |
| 5 | Company Information Guide | A informative chatbot that has a simple company background and allows customers to ask questions about those products. | Text Gen. | Language | Freeform | LLM Prompt |
| 6 | Fictional Captain from the 1700s | Chat with a fictional character from the 1700s without any modern knowledge. | Text Gen. | Language | Freeform | LLM Prompt |
| 7 | Support rep chat summarization | You are a customer support manager and would like to quickly see what your team’s support calls are about. | Summarization | Language | Freeform | LLM Prompt |
| 8 | Summarize news article | News takes too much time to read. You want a quicker way to get the summary. Let Vertex help you. | Summarization | Language | Freeform | LLM Prompt |
| 9 | Chat agent summarization | You are a customer service center manager and you need to quickly see what your agents are talking about. | Summarization | Language | Freeform | LLM Prompt |
| 10 | Chat agent follow up | You are a customer service center manager. Sometimes your agents forget to note down follow ups. You want to automate follow up lists. | Info. Extraction | Language | Freeform | LLM Prompt |
| 11 | Transcript summarization | Summarize a block of text. | Summarization | Language | Structured | LLM Prompt |
| 12 | Dialog summarization | Summarize a conversation. | Summarization | Language | Structured | LLM Prompt |
| 13 | Hashtag tokenization | Create and tokenize hashtags based on the provided text. | Text Gen. | Language | Structured | LLM Prompt |
| 14 | Title generation | Generate a title based on the provided text. | Classification | Language | Structured | LLM Prompt |
| 15 | Sentiment analysis about a person | You would like to see how reporters write about certain people. You have articles and would like to see if a certain person is written about positivly or negatively. | Classification | Language | Freeform | LLM Prompt |
| 16 | Customer request classification, few-shot | Based on customer your customer’s answer, you want to automate routing of your customer to the proper service queue. Use few-shot learning. | Classification | Language | Structured | LLM Prompt |
| 17 | Text classification few-shot | You are an intern at a library and your job is to classify hundreds of articles every day. You’d rather automate this and do something else. | Classification | Language | Structured | LLM Prompt |
| 18 | Article classification | You are an intern at a library and your job is to classify hundreds of articles every day. You’d rather automate this and do something else. | Classification | Language | Freeform | LLM Prompt |
| 19 | Classification headline | Few shot classification on a given topic. | Classification | Language | Structured | LLM Prompt |
| 20 | Sentiment analysis | Explain the sentiment expressed in a body of text. | Classification | Language | Structured | LLM Prompt |
| 21 | Pixel Technical Specifications, one-shot | Generate technical specification from text of a Pixel phone into JSON, one-shot. | Info. Extraction | Language | Structured | LLM Prompt |
| 22 | Wifi troubleshooting | Given description of the different status lights on the Google WiFi router, what should be the troubleshooting step. | Text Gen. | Language | Freeform | LLM Prompt |
| 23 | Contract analysis | You are a partner of a law firm. Your associates are bored of reading contracts to find specific provisions when they can work on more intellectually challenging tasks. | Info. Extraction | Language | Freeform | LLM Prompt |
| 24 | Extractive Question Answering | Answer questions from given background texts. | Text Gen. | Language | Structured | LLM Prompt |
| 25 | Marketing generation Pixel | You work in Google’s device marketing team and you need to create marketing pitch for the new Pixel 7 Pro. You have writers block and need help. | Text Gen. | Language | Freeform | LLM Prompt |
| 26 | Ad copy generation | You are a marketer and want to create different versions of the same ad to target different audiences. You would like some suggestions. | Text Gen. | Language | Freeform | LLM Prompt |
| 27 | Essay outline | Generate an outline for an essay on a particular topic. | Text Gen. | Language | Freeform | LLM Prompt |
| 28 | Correct grammar | Correct grammar in the text. | Text Gen. | Language | Freeform | LLM Prompt |
| 29 | Ad copy from description | Write an ad copy for something based on a description. | Text Gen. | Language | Freeform | LLM Prompt |
| 30 | Write emails and letters | Write an email or letter based on the specified content. | Text Gen. | Language | Freeform | LLM Prompt |
| 31 | Reading comprehension test | Your child is preparing for SAT verbal exam and needs more practice in reading comprehension. | Summarization | Language | Freeform | LLM Prompt |
| 32 | Generate memes | Generate memes based on a certain topic. | Text Gen. | Language | Freeform | LLM Prompt |
| 33 | Interview questions | Generate a list of interview questions targeting a specific position. | Text Gen. | Language | Freeform | LLM Prompt |
| 34 | Naming | Generate ideas for names of a specified entity. | Text Gen. | Language | Freeform | LLM Prompt |
| 35 | General tips and advice | Get tips and advice on general topics. | Text Gen. | Language | Freeform | LLM Prompt |
Conclusion:
The realm of AI has witnessed remarkable advancements, thanks to platforms like Google’s VertexAI. By providing a vast array of pre-built models spanning computer vision, natural language processing, speech processing, and ML tasks on structured tabular data, VertexAI has simplified the development of AI solutions for a multitude of tasks. The platform’s comprehensive selection of models empowers data scientists to efficiently tackle image classification, object detection, sentiment analysis, speech recognition, and much more. Whether it’s creating voice assistants, automating customer support, analyzing visual data, or making data-driven predictions, Vertex AI’s models offer the versatility and performance required to succeed in today’s AI-driven landscape. As AI continues to transform industries, Google’s Vertex AI stands as a powerful tool that unlocks the potential of AI, enabling innovation and driving real-world impact across diverse domains.
By harnessing the power of Vertex AI and its pre-built models, businesses and developers can pave the way for intelligent applications that enhance efficiency, accuracy, and user experiences. With a commitment to ongoing research and development, Google’s Vertex AI is poised to continuously expand its model offerings, ensuring that users have access to cutting-edge AI capabilities and enabling them to push the boundaries of what is possible in the world of artificial intelligence.