# Deployment and operations guide

This guide is for the **Mobile Money edition** of the ISP billing project. Replace every illustrative hostname, path and account name. It is a deployment candidate, not a claim that live payment or router integration has been verified.

## 1. Before you deploy

1. Preserve the original archive. For an existing installation, take a consistent, restorable database backup **and** an encrypted, access-controlled backup of the existing application files, `.env`, uploads and encryption key. Test restoration on a separate staging host. The package contains no customer database or real credentials.
2. Decide whether this is a **new installation** or an **upgrade**. Never use `migrate:fresh`, `db:wipe`, `migrate:refresh` or `--seed` indiscriminately on a production database.
3. Verify the account's PHP web and CLI versions are 8.3+, and enable `bcmath`, `curl`, `dom`, `fileinfo`, `gd`, `intl`, `mbstring`, `pdo`, `pdo_mysql`, `xml`, `zip` and the normal Laravel/OpenSSL/JSON/tokenizer extensions. Verify MySQL/MariaDB, InnoDB, SSH or terminal access, TLS, scheduled tasks, outbound HTTPS to the payment provider and safe connectivity to the routers. Check the exact lock file with `composer check-platform-reqs --no-dev` if Composer is available.
4. Confirm DNS and TLS for the same base host and two subdomains, for example:
   - `https://example.com` — public website and webhook;
   - `https://billing.example.com` — administrator and billing interface;
   - `https://portal.example.com` — customer portal.
   All three must use the same project's `public/` directory as document root. `APP_URL` must be the **base** URL, without a trailing path; do not set it to the `billing.` or `portal.` host. Configure trusted proxies correctly if using a reverse proxy, and only trust proxies controlled by the host. Unknown hosts redirect to the configured base warning page.
5. Keep the application directory, `.env`, `vendor`, `storage`, backups and logs **outside the publicly served document root**. Enable URL rewriting for `public/.htaccess` if applicable. If the host cannot safely point all three hosts to the `public` directory, ask the host for help rather than exposing the project root.
6. Review the original project's licensing inconsistency: `LICENSE` says AGPL-3.0; original Composer metadata says MIT. Preserve notices and clarify rights before redistribution.

## 2. Install a new, empty system

1. Upload/extract the release into a private application directory, for example `/home/ACCOUNT/isp-billing`. Confirm that `artisan`, `public/index.php`, `composer.lock` and `DEPLOYMENT.md` are in that directory. The release includes production Composer dependencies and prebuilt web assets; Node and npm are not required on the host for the initial deployment. If you rebuild the frontend, use a trusted build environment and deploy the generated `public/build` files.
2. Configure all three hosts' document roots to `/home/ACCOUNT/isp-billing/public`, then verify HTTPS and rewrite support. Allow PHP to write only to `storage/` and `bootstrap/cache/`; use appropriate host ownership and restrictive permissions rather than world-writable files. Make sure `.env`, backup files and logs cannot be accessed over HTTP.
3. In the app directory, create a private environment file and edit it:

   ```sh
   cp .env.example .env
   chmod 600 .env
   ```

   Set `APP_URL=https://example.com`, a unique production database/user/password, `APP_ENV=production`, `APP_DEBUG=false`, `APP_TIMEZONE=Africa/Dar_es_Salaam`, HTTPS cookie settings, and suitable mail settings. Leave `MOBILE_MONEY_ENABLED=false` initially. Do not put `.env`, credentials or database backups into a public folder.
4. For a **new** installation only, generate the key, run migrations, seed the non-demo setup data, and create an administrator interactively:

   ```sh
   php artisan key:generate
   php artisan migrate --force
   php artisan db:seed --force
   php artisan app:create-super-admin
   php artisan storage:link
   ```

   Use the correct PHP CLI path on your host. The administrator command prompts for a unique password of at least 12 characters and does not print it. Enable two-factor authentication after login. `storage:link` makes `storage/app/public` publicly accessible: put only intentionally public uploads there, never backups or secrets. Verify the host permits symlinks.
5. Sign in at `https://billing.example.com/login`, open **Site Settings → Billing & Invoicing**, enter `TZS`, tick the acknowledgment that existing values are not converted, and save. Currency is intentionally not guessed by the seeder. Set your actual business name, prices, contact details and SMS templates; remove/replace any inherited demo website content before public launch. Customer signup and authorization settings should be reviewed.
6. Configure the scheduler in section 5. Keep legacy schedules, plugins and the optional queue cron disabled until reviewed. Finally follow the controlled tests in section 7.

## 3. Upgrade an existing installation

1. Work on a separate staging copy first. Preserve the old installation, encryption `APP_KEY`, database, private settings, uploads and a tested restoration point. The old `APP_KEY` is needed for existing encrypted data; **do not run `key:generate` on the upgrade**. The old archive is not itself a database backup.
2. Stop customer payment initiation and coordinate with the payment provider. Inventory and resolve outstanding transactions, pending manual collections, scheduled jobs and router work. Take a final consistent database backup before changing versions. Avoid duplicate schedulers during cutover.
3. Put the new release in a separate directory. Privately copy the existing `.env` and necessary `storage/app/public` uploads, adjusting paths, host, mail and new Mobile Money settings. Do not overwrite existing data with a fresh database. Clear generated caches using the new release's `php artisan optimize:clear` **only after verifying the environment and permissions**; do not copy generated caches between releases. If the project previously used route caching, clear it after changing `APP_URL` and before testing hosts.
4. Compare pending migrations on staging: `php artisan migrate:status`. The release retains a local copy of the old gateway-settings migration so its history is not lost. If a table exists without the corresponding migration record or the schema differs, stop and resolve that discrepancy with a database professional—do not drop the table. Then, on the staged copy and later the backed-up production database, run `php artisan migrate --force`. Do not run `migrate:fresh` on existing data.
5. Review and retain existing accounts, roles, customer records, prices, old payment records and invoice numbers. The revised seeders do not intentionally replace existing currency, permissions or demo users; **do not automatically run seeders on production**. Check for any original default or demo administrator accounts, remove/disable them if not needed, rotate weak or shared passwords, and enable two-factor authentication.
6. The database has a single site currency label. Existing BDT or other historical records and prices are not converted. Do not merely relabel them TZS or combine currencies. With your accountant, choose a proper audited migration or a separate new TZS ledger/installation; retain the historical records and accounting basis. Site Settings requires an acknowledgment before a currency change. Mobile Money checkout refuses to start unless the persisted currency is TZS.
7. Existing historical payment method strings are kept in the database; screens may show a generic **Mobile Money (legacy record)** label. Keep original records accessible for audit. Review all SMS templates, phone numbers and sample website content before use.
8. New backups are stored under `storage/app/private/database-backups` and backup operations require a Super Admin. **Old backups may still exist in public folders** from the original version: inventory, secure, move or delete them only after verifying retention and restore needs. Remove any exposed copies from web access. Existing backup and restore processes must be tested; the host must provide compatible dump/restore binaries (`DB_DUMP_BINARY`/`DB_RESTORE_BINARY` can be set in `.env` if needed).

### Billing and commission policy changes

- Manual and verified online collections use a shared invoice counter. Confirm imported/legacy invoice data and monthly carryover, advance credit, partial payments, free/inactive customers, vouchers and expiry calculations using representative staged accounts. The inherited monthly billing implementation is not a full multi-currency accounting system.
- New reseller commissions default to the **amount actually collected**, rather than a whole package price for every partial payment. This prevents a small partial payment from automatically earning a full-package commission, but it is a business-policy change. Review and agree the policy before cutover. `RESELLER_COMMISSION_BASIS=package` restores package-price calculation for future payments only if that is explicitly intended; it may award a full package commission on each partial payment. Historical commission values are not rewritten.
- Online and manual payment commission records and wallet credit are committed with their payment; a failure rolls back the relevant payment transaction for safe review/retry. Other existing reseller operations and business rules should also be independently tested.

## 4. Configure Mobile Money

Keep checkout disabled until the provider has confirmed a merchant account, endpoint, credentials and test plan. Supply these **server-side** only in `.env`:

```dotenv
MOBILE_MONEY_ENABLED=false
MOBILE_MONEY_API_KEY=your-server-api-key
MOBILE_MONEY_WEBHOOK_SECRET=your-separate-webhook-signing-secret
MOBILE_MONEY_BASE_URL=https://api.sonicpesa.com/api/v1
```

Register this HTTPS webhook in the merchant dashboard:

```text
https://example.com/api/payment/mobile-money/webhook
```

The implementation expects JSON `payment.success` events and `X-SonicPesa-Signature` containing a lowercase hexadecimal HMAC-SHA256 of the **exact raw request body** using the separate webhook secret. It expects the status API to return the order ID, payment status, exact amount, `TZS`, transaction ID, and, when provided, the payer phone. The provider's **current documentation and actual test responses must be checked**; if its contract differs, update and retest the integration before enabling it. Do not bypass signature or server-side status verification to make a test pass.

After editing `.env`, clear stale configuration: `php artisan config:clear`. A config cache may be used only after the final `.env` is in place, and must be rebuilt after changes. Route caching was smoke-tested in the test environment, but is optional; if `APP_URL` or domain configuration changes, clear it and retest every host. Never expose the API key or webhook secret in frontend code, support tickets or logs.

Set `MOBILE_MONEY_ENABLED=true` only after successful controlled tests and approval. This flag blocks **new checkout requests**; existing signed callbacks and reconciliation may still process previously created orders while credentials remain configured. Plan a shutdown with the provider rather than assuming the flag cancels outstanding orders. No automatic refund or settlement verification is implemented.

### Uncertain requests and reconciliation

- An unclear initiation is marked `UNKNOWN` and blocks another request for that customer. Do not tell the customer to pay again until the provider confirms whether an order/prompt exists. A webhook that arrives before its order ID is saved receives a retryable error; scheduled reconciliation also checks pending orders.
- Authorized support can find reference, status and timestamps in `mobile_money_orders`. Protect this information. After at least five minutes, and after independently verifying the provider order ID with support, run:

  ```sh
  php artisan mobile-money:resolve THE-LOCAL-REFERENCE --order=VERIFIED-REMOTE-ORDER-ID
  ```

  The status response must match order, TZS amount and phone. A different or missing phone cannot be linked by this command. If the provider definitively confirms that **no order and no payment prompt exist**, an operator may instead use `--confirmed-no-order` and explicitly answer the confirmation. An incorrect closure can cause a duplicate payment. Never close an uncertain order merely to unblock the customer.
- `php artisan mobile-money:reconcile` and `php artisan mobile-money:retry-router` process bounded batches of three each. At a five-minute cron interval this is at most about 36 checks/actions per hour per command; backlog, network timeouts and provider limits can make it slower. Monitor `UNKNOWN`/old `PENDING` orders, failed logs and pending router actions. Escalate discrepancies; do not manually mark an order successful without verified financial reconciliation.
- A late independently verified success can be recorded even after an earlier cancellation. Reconcile with the provider before issuing another prompt. Router actions are retried independently and do not post another collection. A `DONE` router action means the expected secret state was read back in the supported format, not that a subscriber's end-to-end internet service was independently tested. SMS notifications are best-effort and may require separate monitoring/retry.

## 5. Spaceship shared hosting and cron

Spaceship's published [hosting acceptable use policy](https://www.spaceship.com/legal/hosting-aup/) prohibits cron intervals below five minutes, more than five simultaneous cron jobs, and standalone/unattached server-side daemons; it also limits sustained resource use. Check the current policy and your plan with Spaceship. Their [PHP settings guide](https://www.spaceship.com/knowledgebase/configure-php-settings-spaceship-web-hosting/) and [document-root guidance](https://www.spaceship.com/blog/shared-hosting-by-spaceship-updates/) describe host configuration. The application is resource-intensive and shared hosting may not be suitable for a large ISP or always-on router management.

Use **one** attached cron every five minutes, after confirming that `flock` and the correct PHP CLI are available. Example only—replace both paths:

```cron
*/5 * * * * PHP_BIN=/path/to/php /bin/sh /home/ACCOUNT/isp-billing/bin/run-scheduler.sh >> /home/ACCOUNT/isp-billing/storage/logs/scheduler-cron.log 2>&1
```

The script runs one foreground `schedule:run` protected by `flock`; it does not launch a detached daemon. Check the cron timezone and log output, protect and rotate logs, and ensure the process reliably finishes before the next interval. If `flock` is unavailable, ask the host for a safe non-overlap alternative; do not add an unprotected duplicate cron. Avoid other account cron jobs that violate the account-wide policy.

Default `.env` flags:

```dotenv
QUEUE_CRON_ENABLED=false
LEGACY_BILLING_SCHEDULE_ENABLED=false
LEGACY_ROUTER_SCHEDULE_ENABLED=false
LEGACY_SMS_SCHEDULE_ENABLED=false
```

**Consequences:** automatic monthly bills, scheduled customer disconnections, router sync/polling, hotspot expiry and scheduled SMS are disabled. The inherited jobs scan potentially large datasets and some may exceed a shared-host limit. Review correctness, load-test and obtain host permission before enabling the relevant flags one by one; otherwise use an appropriate hosting arrangement and an operator-approved alternative. Five-minute payment reconciliation and router retries still require the cron above. If no cron runs, those background retries do not run.

Database queues are configured by default but the finite queue runner is off. `QUEUE_CRON_ENABLED=true` adds an attached worker capped at 40 seconds/20 jobs, with a 45-second timeout and one attempt. Enable only if the host approves and jobs fit the limits; monitor failed jobs and beware of duplicate external side effects when retrying. No permanent queue daemon is recommended on this shared plan.

## 6. Router and data security

Use a tightly controlled VPN/private network or verified encrypted management channel with least-privilege RouterOS credentials. Never expose RouterOS plaintext API port 8728 to the public Internet. Confirm firewall, port, certificate/host-key protection and the host's network policy; the inherited router management modules have not been fully security-audited. Restrict access to trusted staff and independently review command construction, escaping, permissions and network topology before enabling them. Test reconnection, expired-profile handling, quoted/special-character inputs, failed commands and retry behavior on a non-production router/customer first. A router timeout must not be treated as proof of activation.

Keep backup files and router credentials private. Restrict database/user privileges, monitor application logs, change default credentials, and confirm the privacy and retention obligations for subscriber data. Plugin loading is disabled unless explicitly enabled with `ENABLED_PLUGINS` and should remain off for untrusted plugins.

## 7. Acceptance checks before real customers

On a separate staging database with approved test credentials:

- Verify base/billing/portal TLS, administrator login, customer login, permissions, currency display, sample prices, public-upload access and denial of `.env`/backup access.
- Confirm local `0…`, `255…` and `+255…` Tanzanian mobile-number formats and invalid input; review pre-existing international numbers rather than silently rewriting them.
- With a small controlled provider test, verify create request, phone prompt, cancellation, pending, success, signed raw-body webhook, status query, exact TZS amount, unique transaction ID, duplicate delivery and reconciliation. Check the provider statement against one collection and the correct balance. Test an incorrect signature, wrong amount/currency/phone and a timeout without creating a second prompt.
- Simulate a router outage and restoration; confirm the payment remains recorded, a pending router action retries, the correct profile is restored, and actual customer access is as intended. Verify monthly billing, due/advance carryover, invoices, voucher use, SMS and reseller commissions with the operator's agreed business rules.
- Confirm cron intervals, non-overlap, job duration and resource use under realistic traffic. Document monitoring, alerting and support escalation. Do not go live merely because automated tests passed.

The frontend build succeeded with warnings about large JavaScript chunks, deprecated Sass syntax and CSS nesting on older browser targets; test the supported browsers and performance for your users.

Automated tests use faked provider responses/router calls. SQLite feature tests and disposable MariaDB concurrent-writer tests were run on PHP 8.4, but do not prove live provider compatibility, settlement, external networks, shared-host capacity, or actual RouterOS behavior.

## 8. Rollback

Before cutover define who authorizes rollback and preserve a synchronized copy of the old code, database, `.env`/`APP_KEY`, uploads and cron settings. Stop new payment initiation and coordinate callbacks with the provider. Back up the **current** database and transaction/provider records before reverting anything; record all payments received since the pre-upgrade snapshot. Restore the old release and its matching configuration/database only under a planned outage and reconcile/reapply post-snapshot transactions with an accountant. A blind database restore can erase legitimate customer payments. Do not roll back/drop the new payment tables while they contain unreconciled payments. Old provider routes and schedules differ; do not re-enable them without security and host review.
