Skip to content

fix: persist keyword changes reliably in backend #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Benevanio merged 1 commit into from
Apr 7, 2026
Merged

fix: persist keyword changes reliably in backend #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions backend/src/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import { existsSync, readFileSync } from "fs";
import path from "path";
import { fileURLToPath } from "url";

const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));

const DEFAULT_KEYWORDS = [
"Java",
Expand Down Expand Up @@ -35,14 +38,29 @@ function parseNumber(value, fallback) {
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}

function getKeywordsFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}

function normalizeKeywords(keywords) {
if (!Array.isArray(keywords)) {
return null;
}

return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
}
Comment on lines +41 to +54

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getKeywordsFilePath()/normalizeKeywords() are duplicated here and in src/jobsApiApp.js. Since the API writes the keywords file and getConfig() reads it, duplicating the path/normalization logic risks divergence and hard-to-debug persistence issues. Consider extracting a shared helper module to keep these rules in one place.

Copilot uses AI. Check for mistakes.

function parseKeywords(value) {
// Tenta pegar do arquivo environment.json
try {
const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");
const envPath = getKeywordsFilePath();
if (existsSync(envPath)) {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
if (Array.isArray(data.KEYWORDS) && data.KEYWORDS.length > 0) {
return data.KEYWORDS;
if (Array.isArray(data.KEYWORDS)) {
return normalizeKeywords(data.KEYWORDS) ?? [];
}
}
} catch (err) {
Expand Down
95 changes: 60 additions & 35 deletions backend/src/jobsApiApp.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,57 @@ import cors from "cors";
import express from "express";
import { existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from "fs";
import path from "path";
import { fileURLToPath } from "url";
import XLSX from "xlsx";
import { run as runScraper } from "./app.js";
import { getConfig } from "./config.js";
import { searchJobsWithCache } from "./pipeline/searchJobsWithCache.js";
import { sources } from "./sources/index.js";

const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));

const DEFAULT_ALLOWED_ORIGINS = [
"https://painel-vagas-lake.vercel.app",
"http://localhost:5173",
"http://localhost:5174",
];

function getKeywordsFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}

function normalizeKeywords(keywords) {
if (!Array.isArray(keywords)) {
return null;
}

return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
}
Comment on lines +20 to +33

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getKeywordsFilePath()/normalizeKeywords() are duplicated here and in src/config.js. Because both the API and getConfig() must agree on the exact keywords file path and normalization rules, duplicating the logic risks future drift (writing to one path while reading from another). Consider extracting these helpers into a shared module that both files import.

Copilot uses AI. Check for mistakes.

function readEnvironmentData() {
const envPath = getKeywordsFilePath();

if (!existsSync(envPath)) {
return { KEYWORDS: [] };
}

try {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
return data && typeof data === "object" ? data : { KEYWORDS: [] };

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readEnvironmentData() treats any parsed JSON with typeof === "object" as valid, which includes arrays. If the file accidentally contains [], the function will return an array and later POST spreads it into an object (adding numeric keys) before writing back. Consider rejecting arrays explicitly (e.g., require a plain object) so corrupted/invalid shapes don't produce surprising persisted output.

Suggested change
return data && typeof data === "object" ? data : { KEYWORDS: [] };
return data && typeof data === "object" && !Array.isArray(data) ? data : { KEYWORDS: [] };

Copilot uses AI. Check for mistakes.
} catch {
return { KEYWORDS: [] };
}
}

function writeEnvironmentData(data) {
const envPath = getKeywordsFilePath();
mkdirSync(path.dirname(envPath), { recursive: true });
writeFileSync(envPath, JSON.stringify(data, null, 2), "utf-8");
}

function parseAllowedOrigins(value) {
const configuredOrigins = String(value ?? "")
.split(",")
Expand Down Expand Up @@ -258,27 +297,20 @@ export function createJobsApiApp(options = {}) {
*/
app.post("/api/keywords", (req, res) => {
try {
const { keywords } = req.body;
const normalizedKeywords = normalizeKeywords(req.body?.keywords);

if (!Array.isArray(keywords)) {
if (normalizedKeywords === null) {
return res.status(400).json({
message: "O campo 'keywords' deve ser um array de strings.",
});
}

const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");
let envData = { KEYWORDS: [] };

if (existsSync(envPath)) {
try {
envData = JSON.parse(readFileSync(envPath, "utf-8"));
} catch (e) {
// Se o arquivo estiver corrompido, reinicia com o objeto padrao
}
}
const envData = {
...readEnvironmentData(),
KEYWORDS: normalizedKeywords,
};

envData.KEYWORDS = keywords;
writeFileSync(envPath, JSON.stringify(envData, null, 2), "utf-8");
writeEnvironmentData(envData);

return res.json({
ok: true,
Expand All @@ -303,28 +335,21 @@ export function createJobsApiApp(options = {}) {
* 200:
* description: Lista de keywords
*/
app.get("/api/keywords", (req, res) => {
try {
const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");

if (!existsSync(envPath)) {
return res.json({ keywords: [] });
}

const fileContent = readFileSync(envPath, "utf-8");
const envData = JSON.parse(fileContent);
app.get("/api/keywords", (_req, res) => {
try {
const envData = readEnvironmentData();

return res.json({
ok: true,
keywords: envData.KEYWORDS || [],
});
} catch (error) {
return res.status(500).json({
message: "Erro ao buscar keywords.",
error: error?.message || "Erro desconhecido",
});
}
});
return res.json({
ok: true,
keywords: normalizeKeywords(envData.KEYWORDS) ?? [],
});
} catch (error) {
return res.status(500).json({
message: "Erro ao buscar keywords.",
error: error?.message || "Erro desconhecido",
});
}
});

/**
* @swagger
Expand Down
19 changes: 19 additions & 0 deletions backend/tests/integration/jobsApi.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ describe("jobs API", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.run.mockResolvedValue(undefined);
delete process.env.KEYWORDS_FILE_PATH;
});

afterEach(() => {
delete process.env.KEYWORDS_FILE_PATH;
tmpDir = undefined;
});

Expand Down Expand Up @@ -179,6 +181,23 @@ describe("jobs API", () => {
});
});

it("POST /api/keywords cria o arquivo configurado e aceita lista vazia", async () => {
tmpDir = mkdtempSync(join(tmpdir(), "jobs-api-"));
process.env.KEYWORDS_FILE_PATH = join(tmpDir, "nested", "environment.json");

const app = createJobsApiApp({ outputDir: tmpDir });
const res = await request(app)
.post("/api/keywords")
.send({ keywords: [" ", ""] })
.expect(200);

expect(res.body).toEqual({
ok: true,
message: "Keywords atualizadas com sucesso.",
keywords: [],
});
});
Comment on lines +184 to +199

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new test says it "cria o arquivo configurado", but it only asserts the HTTP response body. To actually validate the behavior the PR is targeting, add assertions that the file at KEYWORDS_FILE_PATH was created (including nested directories) and that its JSON content contains KEYWORDS: [] after normalization.

Copilot uses AI. Check for mistakes.

it("GET /api/keywords retorna 200 com as keywords", async () => {
const app = createJobsApiApp({ outputDir: tmpDir });
const res = await request(app)
Expand Down
26 changes: 24 additions & 2 deletions backend/tests/unit/utils/config.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import { mkdtempSync, writeFileSync } from "fs";
import { tmpdir } from "os";
import path from "path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { getConfig } from "../../../src/config.js";

Expand All @@ -17,6 +20,7 @@ const CONFIG_ENV_KEYS = [
"JOB_TYPES",
"TIME_FILTER",
"SEARCH_KEYWORDS",
"KEYWORDS_FILE_PATH",
];

describe("getConfig", () => {
Expand Down Expand Up @@ -45,9 +49,12 @@ describe("getConfig", () => {
});

it("parseia SEARCH_KEYWORDS em lista", () => {
vi.stubEnv("SEARCH_KEYWORDS", "Java","Spring","RabbitMQ","Docker");
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
vi.stubEnv("KEYWORDS_FILE_PATH", path.join(tempDir, "missing-environment.json"));
vi.stubEnv("SEARCH_KEYWORDS", "Java,Spring,RabbitMQ,Docker");

Comment on lines 51 to +55

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests create temporary directories via mkdtempSync but never remove them. Over time (especially in local runs), this can leave many stray jobs-config-* folders in the OS temp dir. Consider tracking the created tempDir/keywordsFile per test and cleaning it up in afterEach (e.g., rmSync(tempDir, { recursive: true, force: true })).

Copilot uses AI. Check for mistakes.
const config = getConfig();
expect(config.keywords).toEqual(["Java","Spring","RabbitMQ","Docker"]);
expect(config.keywords).toEqual(["Java", "Spring", "RabbitMQ", "Docker"]);
});

it("rejeita TIME_FILTER invalido e usa fallback", () => {
Expand Down Expand Up @@ -98,9 +105,24 @@ describe("getConfig", () => {
});

it("retorna keywords padrao quando lista informada e vazia", () => {
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
vi.stubEnv("KEYWORDS_FILE_PATH", path.join(tempDir, "missing-environment.json"));
vi.stubEnv("SEARCH_KEYWORDS", " , , ");

const config = getConfig();
expect(config.keywords.length).toBeGreaterThan(0);
expect(config.keywords).toContain("Java");
});

it("preserva lista vazia quando o arquivo de keywords ja existe", () => {
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
const keywordsFile = path.join(tempDir, "environment.json");
writeFileSync(keywordsFile, JSON.stringify({ KEYWORDS: [] }), "utf-8");

vi.stubEnv("KEYWORDS_FILE_PATH", keywordsFile);
vi.stubEnv("SEARCH_KEYWORDS", "Java,Node");

const config = getConfig();
expect(config.keywords).toEqual([]);
});
});
Loading