[Feature] Enhance the quickstart experience

[JinkunLiu]

Motivation

Enhance the quickstart experience. - Currently, there are too many manual steps, thus chances for users to make mistakes and run into problems.

Detail

Import Wizard

Automatically detect and configure common environments, perform safe setup steps, and prompt only for essential choices so users can complete the quickstart with minimal errors.

Proposed workflow:

1. Install Apache Flink (if not already installed)
2. Set Up Python Environment (Python only)
3. Configure PYTHONPATH (Python only)
4. Install Flink Agents and pyFlink Package via pip (Python only)
5. Install Flink Agents Java Library on Flink (Python only)

Optimize Installation Docs

Reorganize installation docs so users can get running quickly via wizard while still providing detailed manual steps for advanced or offline setups.

Proposed catalog structure:

# Overview

# Install Flink Agents
## Wizard (Recommended)
## Manual

# Prepare Your Job
## Maven Dependencies
## Python Environment Check

# Execute Your Job
## From IDE (Minicluster)
## To a Flink cluster

[JinkunLiu]

Hello @xintongsong
I saw you raised a similar idea about enhancing the quickstart experience in #516. Please take a look and let me know if this matches your expectations or if I missed anything. Thanks!

[xintongsong]

Thanks for preparing this, @JinkunLiu .

When I proposed the quickstart experience enhancement, what I had in mind was not just documentation improvements, but to actually reduce the steps users need to take in order to get Flink Agents work.

To be specific, we may provide a one line command (like curl -fsSL https://openclaw.ai/install.sh | bash from OpenClaw ), which brings up an installation wizard, detects the environments, self-configures as much as possible, and asks the user only for necessary decisions.

Regarding your proposal:

Instead of asking users to submit Flink Agents jobs to a standalone Flink cluster, we might suggest running them in MiniCluster. This may be more beginner-friendly and aligns with how Flink’s Quick Start works.

No entirely sure what do you mean by "Flink's Quick Start". Flink's First Steps provides 3 options, none of the uses the MiniCluster.

I personally am not a big fan of Flink's MiniCluster, as it has quite some unique behaviors that are different from other deployment modes and hard to explain.

Provide a archetype Template for user

I like this idea. Just want to mention that, the Flink Agents project is still in its preview versions (0.x) and is rapidly evolving. If we introduce the archetype now, we probably also need to frequently update it w.r.t. the breaking changes. Alternatively, we can wait until the project becomes more stable. I've never worked on a archetype, thus not sure how much effort that takes. WDYT?

Installation Section

I'd suggest the following structure:

# Overview

# Install Flink Agents
## Wizard (Recommended)
## Manual

# Prepare Your Job
## Maven Dependencies (for Java)
## Maybe a few words for Python?

# Execute Your Job
## From IDE (Minicluster)
## To a Flink cluster

[JinkunLiu]

Hello @xintongsong Thanks for your advice.

About you question:

Not entirely sure what you mean by "Flink's Quick Start". Flink's First Steps provides three options, none of which uses the MiniCluster.

I think the Flink-Agents docs Quickstart is analogous to Flink's Tutorial section.
Perhaps my earlier wording was a bit unclear — I only want to use mini cluster in the Flink-Agents docs Quickstart instead of a standalone cluster. I think move Submit Flink Agents Job to Standalone Flink Cluster to another location maybe is better? But If you're concerned about submitting the Quickstart demo to the miniCluster, it's better to keep that.

Alternatively, we can wait until the project becomes more stable. I've never worked on an archetype, thus not sure how much effort that takes. WDYT?

I agree that we can do it later. In terms of development cost, I don't think the overall difficulty is high; however, iterating before things are stable might be a bit troublesome.

I'd suggest the following structure

Yes, that structure looks good, But the reason I want to split the Installation section into mini-cluster and standalone-cluster is that when I first learned Flink I followed the documentation step by step and later found that installing Flink wasn't necessary for the tutorial. So I want improved this at flink-agents.

installation wizard

I want to know whether you have more details about what the Wizard part should do? I have picked some titles from the docs that are in the current version; I'm not sure if they match what you expected.

1. Install Apache Flink (if not already installed)
2. Set Up Python Environment (Python only)
3. Configure PYTHONPATH (Python only)
4. Install Flink Agents and pyFlink Package via pip (Python only)
5. Install Flink Agents Java Library on Flink (Python only)

[xintongsong]

This may be more beginner-friendly and aligns with how Flink’s Quick Start works.

I think the Flink-Agents docs Quickstart is analogous to Flink's Tutorial section.

I also don't find how the tutorial you mentioned suggests using a mini cluster. I'm asking because you said using mini cluster is more aligned with Flink's quick start, which sounds like there's a quick start that suggests using mini cluster.

I only want to use mini cluster in the Flink-Agents docs Quickstart instead of a standalone cluster.

I hold a completely opposite view. I think we should avoid suggesting mini cluster as much as possible. This comes from years of experiences supporting Flink users. Yes, it requires less steps than setting up a standalone cluster, but the gap is very small. On the other hand, many behaviors of mini cluster is quite different from other deployment modes. Many features of Flink do not work properly on MiniCluster, due to technical limitations, of developers' lack of attention on mini cluster. As a result, when user trying out things on Flink, there's a much higher chance of running into unexpected problems, leading to a worse experience.

I want to know whether you have more details about what the Wizard part should do? I have picked some titles from the docs that are in the current version; I'm not sure if they match what you expected.

1. Install Apache Flink (if not already installed)
2. Set Up Python Environment (Python only)
3. Configure PYTHONPATH (Python only)
4. Install Flink Agents and pyFlink Package via pip (Python only)
5. Install Flink Agents Java Library on Flink (Python only)

Yes, these are all good fits for the wizard. Ideally, users only need to make decisions / choices. Everything else that requires manual operations, the wizard should do it for the users. We can do this step-by-step. There's no need to cover everything in the first step.

[JinkunLiu]

Thanks for explaining that.

This may be more beginner-friendly and aligns with how Flink’s Quick Start works.

I based this on the Running the Application section of the Flink Tutorial. At Tutorial only shows how to run flink app in IDE.

I clearly understand the concerns you raised now. and I support guiding users to submit the Flink job to a standalone cluster to avoid various unexpected problem.

[xintongsong]

I based this on the Running the Application section of the Flink Tutorial. At Tutorial only shows how to run flink app in IDE.

Thanks for the pointer. I didn't notice it.

I clearly understand the concerns you raised now. and I support guiding users to submit the Flink job to a standalone cluster to avoid various unexpected problem.

Great. Seems we are aligned now. Would you like to update the plan accordingly, and we can move to the doc PR after a quick double-check.

[wenjin272]

Thanks for this proposal, @JinkunLiu

Instead of asking users to submit Flink Agents jobs to a standalone Flink cluster, we might suggest running them in MiniCluster. This may be more beginner-friendly and aligns with how Flink’s Quick Start works.

I agree with the viewpoint of @xintongsong. In addition to the reasons he mentioned, certain features of Flink-Agents, such as event logs and metrics, are easier to display when a web UI is available.

[JinkunLiu]

Thanks for you remind @wenjin272

Yes — I understand your concern. Would it be acceptable to use MiniCluster only for the walkthrough? Actually we can run localy with web UI via calling createLocalEnvironmentWithWebUI and add theflink-runtime-web dependency to the project's pom.xml

[wenjin272]

I think anyway we need to retain the instructions on how to submit a Flink-Agents jobs to a standalone Flink cluster.

I agree the mini cluster is friendly to developing and testing. Maybe we can extract a dedicated section to show how to develop a flink-agents job on IDE and mini cluster, including the code walkthrough. And in the quick start section, we provide a one line command for supporting user submit the agent job to standalone cluster. WDYT @xintongsong.

[JinkunLiu]

Yes — absolutely. Retaining the instructions for how to submit Flink-Agents jobs to a standalone Flink cluster is very important.

Perhaps I didn't make myself clear: what I meant is that I would like to remove it from the current Quick Start section. Maybe we should move it to another location. I think the Quick Start section should mainly help users learn the API and get started quickly, so it should be as easy as possible.

[wenjin272]

I think the Quick Start section should mainly help users learn the API and get started quickly, so it should be as easy as possible.

I think there is no fundamental disagreement between us; rather, we are emphasizing different priorities.

1. Simply the Code Walkthrough, like providing archetype/template and running on mini cluster to help users learn how to develop a flink-agents job quickly.
2. Simply the Run Example like providing an installation script to help users submit a exist flink-agents job to a flink standalone cluster easily.

In our 0.3 plan, we focused more on the second point, as we observed that some users encountered a less smooth experience when submitting jobs to the Flink cluster due to the numerous steps and configuration options involved. However, I believe the first point you mentioned is also important, as our familiarity with developing Flink Agents jobs may have caused us to overlook this aspect.

As far as I'm concerned, I think we need to improve both the two points. Regarding whether to move the instructions for submitting Flink-Agents jobs to a standalone Flink cluster into a dedicated section, I believe either approach is acceptable.

Let's also wait for xintong's opinion.

[xintongsong]

Nice discussion. I think separating the two aspects of "code walkthrough" and "execute an agent" makes sense. With the one-line wizard, it should be good enough for people who only interested in quickly learning the apis to skip the unnecessary installation details.