{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cell_style": "center"
   },
   "outputs": [],
   "source": [
    "# Copyright 2026 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RMI Tutorial 04: Pub/Sub Real-time Verification\n",
    "\n",
    "RMI provides near real-time traffic updates via Cloud Pub/Sub. This tutorial uses the `gcloud` CLI to verify your real-time data stream.\n",
    "\n",
    "### Objectives:\n",
    "1. **Identify** your RMI provider topic.\n",
    "2. **Create** a local Pull subscription.\n",
    "3. **Retrieve** and parse real-time messages via the CLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/googlemaps-samples/insights-samples/blob/main/roads_management_insights/tutorials/04_pubsub_verification.ipynb) \n",
    "[![Open In Colab Enterprise](https://img.shields.io/badge/Open%20in-Colab%20Enterprise-blue?logo=google-cloud&logoColor=white)](https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fgooglemaps-samples%2Finsights-samples%2Fmain%2Froads_management_insights%2Ftutorials%2F04_pubsub_verification.ipynb) \n",
    "[![Open In BigQuery Notebooks](https://img.shields.io/badge/Open%20in-BigQuery%20Notebooks-blue?logo=google-cloud&logoColor=white)](https://console.cloud.google.com/bigquery/import?url=https%3A%2F%2Fraw.githubusercontent.com%2Fgooglemaps-samples%2Finsights-samples%2Fmain%2Froads_management_insights%2Ftutorials%2F04_pubsub_verification.ipynb) \n",
    "[![View on GitHub](https://img.shields.io/badge/View%20on-GitHub-lightgrey?logo=github&logoColor=white)](https://github.com/googlemaps-samples/insights-samples/blob/main/roads_management_insights/tutorials/04_pubsub_verification.ipynb)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Authenticate with Google Cloud\n",
    "from google.colab import auth\n",
    "auth.authenticate_user()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Environment Setup\n",
    "Set your Project ID and Project Number (from Tutorial 01)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_ID = \"YOUR_PROJECT_ID\" # @param {type:\"string\"}\n",
    "PROJECT_NUMBER = \"YOUR_PROJECT_NUMBER\" # @param {type:\"string\"}\n",
    "SUBSCRIPTION_ID = \"tutorial-rmi-json-pull-sub\"\n",
    "\n",
    "import os\n",
    "os.environ[\"PROJECT_ID\"] = PROJECT_ID\n",
    "# Note: Topic names are provided by the RMI team and may vary.\n",
    "os.environ[\"TOPIC_PATH\"] = f\"projects/maps-platform-roads-management/topics/roadsinformation-{PROJECT_NUMBER}-json\"\n",
    "os.environ[\"SUBSCRIPTION_PATH\"] = f\"projects/{PROJECT_ID}/subscriptions/{SUBSCRIPTION_ID}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Create the Pull Subscription\n",
    "We create a subscription in your local project to listen to the RMI provider topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud pubsub subscriptions create \"${SUBSCRIPTION_PATH}\" \\\n",
    "  --topic=\"${TOPIC_PATH}\" \\\n",
    "  --project=\"${PROJECT_ID}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Pull and Parse Messages\n",
    "Retrieve messages from the stream. We use `--auto-ack` to automatically acknowledge receipt and `--format=\"json\"` to view the full RMI payload."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud pubsub subscriptions pull \"${SUBSCRIPTION_PATH}\" \\\n",
    "  --limit=5 \\\n",
    "  --auto-ack \\\n",
    "  --format=\"json\" \\\n",
    "  --project=\"${PROJECT_ID}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Optional: Create a BigQuery Subscription\n",
    "For long-term storage and analysis, you can stream RMI updates directly into a BigQuery table.\n",
    "\n",
    "### When to use a BigQuery Subscription vs. RMI Shared Tables?\n",
    "RMI already provides a **`recent_roads_data`** table via your Analytics Hub subscription. \n",
    "\n",
    "**Use the RMI Shared Table (`recent_roads_data`) if:**\n",
    "- You only need the last 24 hours of data.\n",
    "- You want a zero-maintenance, managed experience.\n",
    "\n",
    "**Use a BigQuery Subscription (Custom) if:**\n",
    "- You need to **retain** data for more than 24 hours in a raw format.\n",
    "- You want to perform **custom real-time transformations** during ingestion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Target Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "bq mk --project_id=${PROJECT_ID} --dataset rmi_realtime_stream"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grant Permissions to Pub/Sub Service Account\n",
    "Pub/Sub requires the `BigQuery Data Editor` role on your project (or dataset) to write messages into the table. The service account follows the pattern: `service-PROJECT_NUMBER@gcp-sa-pubsub.iam.gserviceaccount.com`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud projects add-iam-policy-binding ${PROJECT_ID} \\\n",
    "    --member=\"serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com\" \\\n",
    "    --role=\"roles/bigquery.dataEditor\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the BigQuery Subscription\n",
    "This creates a no-code streaming pipeline from Pub/Sub to BigQuery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "gcloud pubsub subscriptions create \"projects/${PROJECT_ID}/subscriptions/tutorial-rmi-bq-sub\" \\\n",
    "  --topic=\"${TOPIC_PATH}\" \\\n",
    "  --bigquery-table=\"${PROJECT_ID}:rmi_realtime_stream.realtime_updates\" \\\n",
    "  --use-topic-schema \\\n",
    "  --project=\"${PROJECT_ID}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Optional: Subscription Filtering\n",
    "You can use [Subscription Filters](https://docs.cloud.google.com/pubsub/docs/subscription-message-filter) to only receive a subset of messages.\n",
    "\n",
    "### Create a Filtered Subscription\n",
    "Only receives messages for a specific route ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# Create a subscription with prefix filtering\n",
    "# This only receives messages where the route ID starts with \"PREFIX-\"\n",
    "gcloud pubsub subscriptions create \"projects/${PROJECT_ID}/subscriptions/tutorial-rmi-prefix-filtered-sub\" \\\n",
    "  --topic=\"${TOPIC_PATH}\" \\\n",
    "  --message-filter='attributes.selected_route_id.hasPrefix(\"PREFIX-\")' \\\n",
    "  --project=\"${PROJECT_ID}\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary & Next Steps\n",
    "You have successfully verified your real-time data stream using the `gcloud` CLI.\n",
    "\n",
    "**Next**: In the next tutorial, we will **Visualize** our RMI data on an interactive map using **pydeck**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**For more advanced RMI query patterns, visit the official [RMI Sample Queries Repository](https://github.com/googlemaps-samples/insights-samples/tree/main/roads_management_insights/rmi_sample_queries).**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}