Skip to content

Develop #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Benevanio merged 2 commits into from
Apr 7, 2026
Merged

Develop #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4e0b56f
fix: persist keyword changes reliably in backend
Benevanio Apr 7, 2026
9071ee2
fix: persist keyword changes reliably in backend (#72)
Benevanio Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions backend/src/config.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
import { existsSync, readFileSync } from "fs";
import path from "path";
import { fileURLToPath } from "url";

const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));

const DEFAULT_KEYWORDS = [
"Java",
Expand Down Expand Up @@ -35,14 +38,29 @@ function parseNumber(value, fallback) {
return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback;
}

function getKeywordsFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}

function normalizeKeywords(keywords) {
if (!Array.isArray(keywords)) {
return null;
}

return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
}

function parseKeywords(value) {
// Tenta pegar do arquivo environment.json
try {
const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");
const envPath = getKeywordsFilePath();
if (existsSync(envPath)) {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
if (Array.isArray(data.KEYWORDS) && data.KEYWORDS.length > 0) {
return data.KEYWORDS;
if (Array.isArray(data.KEYWORDS)) {
return normalizeKeywords(data.KEYWORDS) ?? [];
}
Comment on lines +62 to 64

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parseKeywords only uses the file when data.KEYWORDS is an array; if the file exists but KEYWORDS has an invalid type, it falls back to env/defaults. Meanwhile, the API's GET /api/keywords will return an empty list for the same invalid content. Consider aligning the behavior (e.g., treat "file exists but KEYWORDS invalid" as an empty list everywhere, or reuse a shared read/normalize helper).

Suggested change
if (Array.isArray(data.KEYWORDS)) {
return normalizeKeywords(data.KEYWORDS) ?? [];
}
return normalizeKeywords(data?.KEYWORDS) ?? [];

Copilot uses AI. Check for mistakes.
}
} catch (err) {
Expand Down
95 changes: 60 additions & 35 deletions backend/src/jobsApiApp.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,57 @@ import cors from "cors";
import express from "express";
import { existsSync, mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from "fs";
import path from "path";
import { fileURLToPath } from "url";
import XLSX from "xlsx";
import { run as runScraper } from "./app.js";
import { getConfig } from "./config.js";
import { searchJobsWithCache } from "./pipeline/searchJobsWithCache.js";
import { sources } from "./sources/index.js";

const MODULE_DIR = path.dirname(fileURLToPath(import.meta.url));

const DEFAULT_ALLOWED_ORIGINS = [
"https://painel-vagas-lake.vercel.app",
"http://localhost:5173",
"http://localhost:5174",
];

function getKeywordsFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}

function normalizeKeywords(keywords) {
if (!Array.isArray(keywords)) {
return null;
}

return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normalizeKeywords coerces every array item with String(...), so non-string values (e.g. numbers/objects) are silently accepted even though the 400 message says the field must be an array of strings. Consider validating that every element is a string (and optionally rejecting non-primitive values) before normalizing, or adjust the error contract/message to match the coercion behavior.

Suggested change
return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
if (!keywords.every((item) => typeof item === "string")) {
return null;
}
return [...new Set(keywords.map((item) => item.trim()).filter(Boolean))];

Copilot uses AI. Check for mistakes.
}

function readEnvironmentData() {
const envPath = getKeywordsFilePath();

if (!existsSync(envPath)) {
return { KEYWORDS: [] };
}

try {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
return data && typeof data === "object" ? data : { KEYWORDS: [] };
} catch {
return { KEYWORDS: [] };
}
}

function writeEnvironmentData(data) {
const envPath = getKeywordsFilePath();
mkdirSync(path.dirname(envPath), { recursive: true });
writeFileSync(envPath, JSON.stringify(data, null, 2), "utf-8");
Comment on lines +20 to +53

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The keyword file path resolution + keyword normalization logic is duplicated here and in backend/src/config.js (and there is additional keyword normalization elsewhere). This increases the risk of behavior drifting (e.g., different handling of invalid file content). Consider extracting a single shared helper (e.g. src/keywordsStorage.js) used by both the API and getConfig().

Suggested change
function getKeywordsFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}
function normalizeKeywords(keywords) {
if (!Array.isArray(keywords)) {
return null;
}
return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
}
function readEnvironmentData() {
const envPath = getKeywordsFilePath();
if (!existsSync(envPath)) {
return { KEYWORDS: [] };
}
try {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
return data && typeof data === "object" ? data : { KEYWORDS: [] };
} catch {
return { KEYWORDS: [] };
}
}
function writeEnvironmentData(data) {
const envPath = getKeywordsFilePath();
mkdirSync(path.dirname(envPath), { recursive: true });
writeFileSync(envPath, JSON.stringify(data, null, 2), "utf-8");
function createKeywordsStorage() {
function getFilePath() {
const configuredPath = process.env.KEYWORDS_FILE_PATH?.trim();
return configuredPath
? path.resolve(configuredPath)
: path.resolve(MODULE_DIR, "db", "environment.json");
}
function normalize(keywords) {
if (!Array.isArray(keywords)) {
return null;
}
return [...new Set(keywords.map((item) => String(item ?? "").trim()).filter(Boolean))];
}
function read() {
const envPath = getFilePath();
if (!existsSync(envPath)) {
return { KEYWORDS: [] };
}
try {
const data = JSON.parse(readFileSync(envPath, "utf-8"));
return data && typeof data === "object" ? data : { KEYWORDS: [] };
} catch {
return { KEYWORDS: [] };
}
}
function write(data) {
const envPath = getFilePath();
mkdirSync(path.dirname(envPath), { recursive: true });
writeFileSync(envPath, JSON.stringify(data, null, 2), "utf-8");
}
return {
getFilePath,
normalize,
read,
write,
};
}
const keywordsStorage = createKeywordsStorage();
function getKeywordsFilePath() {
return keywordsStorage.getFilePath();
}
function normalizeKeywords(keywords) {
return keywordsStorage.normalize(keywords);
}
function readEnvironmentData() {
return keywordsStorage.read();
}
function writeEnvironmentData(data) {
keywordsStorage.write(data);

Copilot uses AI. Check for mistakes.
}

function parseAllowedOrigins(value) {
const configuredOrigins = String(value ?? "")
.split(",")
Expand Down Expand Up @@ -258,27 +297,20 @@ export function createJobsApiApp(options = {}) {
*/
app.post("/api/keywords", (req, res) => {
try {
const { keywords } = req.body;
const normalizedKeywords = normalizeKeywords(req.body?.keywords);

if (!Array.isArray(keywords)) {
if (normalizedKeywords === null) {
return res.status(400).json({
message: "O campo 'keywords' deve ser um array de strings.",
});
}

const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");
let envData = { KEYWORDS: [] };

if (existsSync(envPath)) {
try {
envData = JSON.parse(readFileSync(envPath, "utf-8"));
} catch (e) {
// Se o arquivo estiver corrompido, reinicia com o objeto padrao
}
}
const envData = {
...readEnvironmentData(),
KEYWORDS: normalizedKeywords,
};

envData.KEYWORDS = keywords;
writeFileSync(envPath, JSON.stringify(envData, null, 2), "utf-8");
writeEnvironmentData(envData);

return res.json({
ok: true,
Expand All @@ -303,28 +335,21 @@ export function createJobsApiApp(options = {}) {
* 200:
* description: Lista de keywords
*/
app.get("/api/keywords", (req, res) => {
try {
const envPath = path.resolve(process.cwd(), "src", "db", "environment.json");

if (!existsSync(envPath)) {
return res.json({ keywords: [] });
}

const fileContent = readFileSync(envPath, "utf-8");
const envData = JSON.parse(fileContent);
app.get("/api/keywords", (_req, res) => {
try {
const envData = readEnvironmentData();

return res.json({
ok: true,
keywords: envData.KEYWORDS || [],
});
} catch (error) {
return res.status(500).json({
message: "Erro ao buscar keywords.",
error: error?.message || "Erro desconhecido",
});
}
});
return res.json({
ok: true,
keywords: normalizeKeywords(envData.KEYWORDS) ?? [],
});
} catch (error) {
return res.status(500).json({
message: "Erro ao buscar keywords.",
error: error?.message || "Erro desconhecido",
});
}
});

/**
* @swagger
Expand Down
19 changes: 19 additions & 0 deletions backend/tests/integration/jobsApi.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ describe("jobs API", () => {
beforeEach(() => {
vi.clearAllMocks();
mocks.run.mockResolvedValue(undefined);
delete process.env.KEYWORDS_FILE_PATH;
});

afterEach(() => {
delete process.env.KEYWORDS_FILE_PATH;
tmpDir = undefined;
});

Expand Down Expand Up @@ -179,6 +181,23 @@ describe("jobs API", () => {
});
});

it("POST /api/keywords cria o arquivo configurado e aceita lista vazia", async () => {
tmpDir = mkdtempSync(join(tmpdir(), "jobs-api-"));
process.env.KEYWORDS_FILE_PATH = join(tmpDir, "nested", "environment.json");

const app = createJobsApiApp({ outputDir: tmpDir });
const res = await request(app)
.post("/api/keywords")
.send({ keywords: [" ", ""] })
.expect(200);

expect(res.body).toEqual({
ok: true,
message: "Keywords atualizadas com sucesso.",
keywords: [],
});
});
Comment on lines +184 to +199

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test claims it "cria o arquivo configurado" but it only asserts on the HTTP response. As written, it would pass even if the file was never created/written. Either add assertions that the file exists (and contains the expected JSON) or rename the test to reflect what is actually being verified.

Copilot uses AI. Check for mistakes.

it("GET /api/keywords retorna 200 com as keywords", async () => {
const app = createJobsApiApp({ outputDir: tmpDir });
const res = await request(app)
Expand Down
26 changes: 24 additions & 2 deletions backend/tests/unit/utils/config.test.js
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import { mkdtempSync, writeFileSync } from "fs";
import { tmpdir } from "os";
import path from "path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { getConfig } from "../../../src/config.js";

Expand All @@ -17,6 +20,7 @@ const CONFIG_ENV_KEYS = [
"JOB_TYPES",
"TIME_FILTER",
"SEARCH_KEYWORDS",
"KEYWORDS_FILE_PATH",
];

describe("getConfig", () => {
Expand Down Expand Up @@ -45,9 +49,12 @@ describe("getConfig", () => {
});

it("parseia SEARCH_KEYWORDS em lista", () => {
vi.stubEnv("SEARCH_KEYWORDS", "Java","Spring","RabbitMQ","Docker");
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
vi.stubEnv("KEYWORDS_FILE_PATH", path.join(tempDir, "missing-environment.json"));
vi.stubEnv("SEARCH_KEYWORDS", "Java,Spring,RabbitMQ,Docker");

const config = getConfig();
expect(config.keywords).toEqual(["Java","Spring","RabbitMQ","Docker"]);
expect(config.keywords).toEqual(["Java", "Spring", "RabbitMQ", "Docker"]);
});
Comment on lines 51 to 58

Copilot AI Apr 7, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests create temp directories/files via mkdtempSync/writeFileSync but never remove them, which can leave artifacts in the OS temp directory across repeated runs. Consider tracking created paths and cleaning them up in afterEach (e.g., rmSync(tempDir, { recursive: true, force: true })).

Copilot uses AI. Check for mistakes.

it("rejeita TIME_FILTER invalido e usa fallback", () => {
Expand Down Expand Up @@ -98,9 +105,24 @@ describe("getConfig", () => {
});

it("retorna keywords padrao quando lista informada e vazia", () => {
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
vi.stubEnv("KEYWORDS_FILE_PATH", path.join(tempDir, "missing-environment.json"));
vi.stubEnv("SEARCH_KEYWORDS", " , , ");

const config = getConfig();
expect(config.keywords.length).toBeGreaterThan(0);
expect(config.keywords).toContain("Java");
});

it("preserva lista vazia quando o arquivo de keywords ja existe", () => {
const tempDir = mkdtempSync(path.join(tmpdir(), "jobs-config-"));
const keywordsFile = path.join(tempDir, "environment.json");
writeFileSync(keywordsFile, JSON.stringify({ KEYWORDS: [] }), "utf-8");

vi.stubEnv("KEYWORDS_FILE_PATH", keywordsFile);
vi.stubEnv("SEARCH_KEYWORDS", "Java,Node");

const config = getConfig();
expect(config.keywords).toEqual([]);
});
});
Loading